A machine learning model must be judged on data it has never seen. If you test a model on the same data it learned from, the score only measures memorization, not the ability to generalize to new data. Data splitting divides your dataset into separate parts used for training, tuning and final evaluation.
Overfitting and generalization
- Overfitting: the model performs very well on training data but poorly on new data because it memorized noise.
- Underfitting: the model is too simple and performs poorly everywhere.
- Generalization: performing well on unseen data — this is the real goal.
Only a separate, untouched dataset can reveal whether a model generalizes.
The three sets
| Set | Purpose | Typical share |
|---|---|---|
| Training set | The model learns its parameters from it | 60-80% |
| Validation set | Tune hyperparameters, compare models, early stopping | 10-20% |
| Test set | One-time, final, unbiased performance estimate | 10-20% |
Common split ratios
| Ratio (train/val/test) | When to use |
|---|---|
| 80 / 0 / 20 | Small or medium data, with cross-validation on the training part |
| 70 / 15 / 15 | The classic balanced choice |
| 60 / 20 / 20 | When lots of tuning is needed |
| 98 / 1 / 1 | Very large datasets (millions of rows) |
The test set is like a final exam. If you keep peeking at it while building the model, it stops being an honest exam.
Golden rules
- Keep the test set locked away until the very end.
- Never tune hyperparameters on the test set.
- Every preprocessing step that learns from data is fitted on the training set only.