Avoiding Data Leakage and Common Mistakes - Splitting Data | Data Preparation

Leakage is the number one reason a model looks great in testing and fails in production. Splitting correctly is the first defence.

The correct order

Remove duplicates and fix obvious errors.
Split the data.
Fit imputers, scalers, encoders, feature selectors and resamplers on the training set only.
Transform the validation and test sets with the fitted objects.

Wrong vs. right

# WRONG: the scaler sees the test data
from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)                 # fitted on ALL rows
X_train, X_test = train_test_split(X_scaled, test_size=0.2)  # leakage!

# RIGHT: split first, fit on train only
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler().fit(X_train)     # learn mean/std from train
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)        # reuse the same mean/std

Use a pipeline to do it automatically

from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), LogisticRegression(max_iter=1000))
pipe.fit(X_train, y_train)
print("Test accuracy:", pipe.score(X_test, y_test))

Common mistakes

Mistake	Consequence
Scaling or imputing the whole dataset before splitting	Test statistics leak into training
Not stratifying imbalanced data	The test set may contain almost no minority samples
Random split on time-series data	Future information leaks into the past
Tuning hyperparameters on the test set	Overly optimistic final score
Forgetting `random_state`	Results cannot be reproduced
Duplicates in both train and test	The model is tested on rows it has already seen
Features derived from the target	The model "cheats"
Oversampling before splitting	Synthetic copies of test rows land in training

Checklist for a good split

Duplicates removed before splitting
stratify used for classification
Time order respected for time-series data
Group entities kept together
Preprocessing fitted only on training data
Test set used exactly once, at the end
random_state set and documented