Model Evaluation is an important part of building any machine-learning model. The complete dataset is divided into training and testing sets, the model is trained on the training set and its performance is evaluated on the testing set.
Usually, the subset of data used for training and testing is constant. A model can lose stability and its performance may degrade if there is a change in training and testing data. A good stable model is one that is trained so well to perform well with changes in data.
Cross-validation allows us to shuffle the input dataset between training and testing to better evaluate the stability of the trained model on unseen data.
What is Cross-Validation?
Cross-Validation is a technique for evaluating ML models by training several ML models on a subset of data and evaluating them on the complementary subset of the data.
Types of Cross-Validation
There are four types of cross-validation methods:
1. Hold Out Method
- This is the simplest approach in which data is shuffled randomly and then split into training and testing data usually in the ratio of 80:20 or 70:30.
- It can give different results when trained on different shuffled data and cause instability.
- In the smaller datasets, the data selected for training may not be a good representation of complete data which leads to bad performance on unseen data.
- This model works well in the case of large datasets but is usually not so well with a smaller datasets.
2. K-Fold Cross Validation
- In this method, the complete dataset is split into k number of folds.
- For each fold the model is trained on data from k-1 folds and the data from the kth fold is used for testing.
- The final accuracy is calculated by taking the average of the accuracy score from the individual models.
- It guarantees that the final score of the model is not dependent on the way we picked the data.
3. Stratified K-Fold Cross Validation
- In the case of classification models, using k-fold cross-validation can be a bad choice as it can lead to highly imbalanced folds.
- If the training data gets mostly data from one class, the model build will be biased.
- Stratified k-fold cross-validation uses a process of rearranging the data in such a way that each fold is a good representation of complete data.
4. Leave P Out Cross Validation
- In this type of cross-validation, p data points are taken out from the complete dataset where the total number of data points is n.
- n-p data points are used to train the model and p data points are used for testing.
- This process is repeated for all possible p values from the dataset.
5. Leave One Out Cross Validation
- This is a simpler form of leave p out cross validation where p is 1.
- One data point is taken out and the model is trained on n-1 data points where n is the total number of data points.
- This process is repeated keeping every data point as testing data and training model on all other data.
Advantages of Cross-Validation
Cross-validation ensures that model accuracy is checked on different combinations of data as training and testing. It helps to finally select a generalized model which performs well on unseen data.