A random split is wrong for some kinds of data. Two common cases are time-ordered data and data where one entity appears in many rows.
Time-series data
If you shuffle time-ordered data, the model learns from the future to predict the past. Always train on earlier data and test on later data.
Simple chronological split
df_ts = df.sort_values("date") # make sure it is ordered by time
cut = int(len(df_ts) * 0.8)
train = df_ts.iloc[:cut] # the past
test = df_ts.iloc[cut:] # the future
Expanding-window cross-validation
TimeSeriesSplit makes several train/test pairs where the training window keeps growing and the test window always comes after it.
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for fold, (train_idx, test_idx) in enumerate(tscv.split(X), 1):
print(f"Fold {fold}: train 0-{train_idx[-1]} | test {test_idx[0]}-{test_idx[-1]}")
Split 1: TRAIN [0-99] TEST [100-199]
Split 2: TRAIN [0-199] TEST [200-299]
Split 3: TRAIN [0-299] TEST [300-399]
Group-aware splits
If one patient, customer or device appears in many rows, all of its rows must stay in the same split. Otherwise the model can "recognize" a person seen in training and the score becomes unrealistically high.
from sklearn.model_selection import GroupShuffleSplit
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=df["PatientID"]))
X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
Use GroupKFold for cross-validation with groups.
from sklearn.model_selection import GroupKFold, cross_val_score
scores = cross_val_score(model, X, y, cv=GroupKFold(n_splits=5), groups=df["PatientID"])
print(scores.mean())
When to use which split
| Data type | Recommended split |
|---|---|
| Independent rows, classification | Stratified split / Stratified K-Fold |
| Independent rows, regression | Random split / K-Fold |
| Time series, forecasting | Chronological split / TimeSeriesSplit |
| Repeated measurements per subject | GroupShuffleSplit / GroupKFold |
| Images or text with duplicates or near-duplicates | Deduplicate first, then split |
Extra care
- Leave a gap between train and test for time series if features use rolling windows.
- Split by entity (user, patient) rather than by row when the model will be used on new entities.