Cross-Validation

A single validation split can be lucky or unlucky, especially with small data. Cross-validation (CV) gives a more reliable estimate by using every sample for both training and validation.

How k-fold works

1. Shuffle the training data and split it into k folds (commonly 5 or 10).
2. Train on k-1 folds and validate on the remaining fold.
3. Repeat k times so each fold is the validation fold once.
4. Average the k scores.

Fold 1: [VAL][TRAIN][TRAIN][TRAIN][TRAIN]
Fold 2: [TRAIN][VAL][TRAIN][TRAIN][TRAIN]
Fold 3: [TRAIN][TRAIN][VAL][TRAIN][TRAIN]
Fold 4: [TRAIN][TRAIN][TRAIN][VAL][TRAIN]
Fold 5: [TRAIN][TRAIN][TRAIN][TRAIN][VAL]

Cross-validation in scikit-learn

from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(random_state=42)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X_train.fillna(0), y_train, cv=skf, scoring="accuracy")

print("Fold scores:", scores.round(3))
print("Mean: %.3f (+/- %.3f)" % (scores.mean(), scores.std()))

The standard deviation tells you how stable the model is across folds.

Cross-validation techniques

Technique	When to use
Hold-out	Large datasets, quick experiments
K-Fold	Small or medium data, regression
Stratified K-Fold	Classification, especially imbalanced classes
Repeated K-Fold	Even more stable estimates
Leave-One-Out (LOO)	Very tiny datasets (expensive)
Group K-Fold	Several rows per person, patient or device
TimeSeriesSplit	Time-ordered data

Cross-validation with a pipeline

Putting preprocessing inside the pipeline means it is re-fitted inside every fold, preventing leakage.

from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = make_pipeline(
    SimpleImputer(strategy="median"),
    StandardScaler(),
    LogisticRegression(max_iter=1000),
)
scores = cross_val_score(pipe, X_train, y_train, cv=5)
print(scores.mean())

Nested use: tuning + evaluation

from sklearn.model_selection import GridSearchCV

param_grid = {"logisticregression__C": [0.01, 0.1, 1, 10]}
grid = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy")
grid.fit(X_train, y_train)

print("Best params:", grid.best_params_)
print("CV score:", round(grid.best_score_, 3))
print("Test score:", round(grid.score(X_test, y_test), 3))   # final, untouched

GridSearchCV uses cross-validation on the training set to pick the best hyperparameters, and the untouched test set gives the final estimate.