Building a Preprocessing Pipeline

Doing preprocessing by hand for train, test and future data invites mistakes. scikit-learn's Pipeline and ColumnTransformer chain the steps so that exactly the same transformations are learned from training data and applied everywhere else.

Step 1 — split first

from sklearn.model_selection import train_test_split

X = df[["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]]
y = df["Survived"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

Step 2 — define transformations per column type

• Numeric columns: fill missing values with the median, then standardize.
• Categorical columns: fill missing values with the most frequent value, then one-hot encode.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_features = ["Age", "SibSp", "Parch", "Fare"]
categorical_features = ["Pclass", "Sex", "Embarked"]

numeric_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

categorical_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipe, numeric_features),
    ("cat", categorical_pipe, categorical_features),
])

Step 3 — fit on train, transform both

X_train_ready = preprocessor.fit_transform(X_train)  # learns medians, means, categories
X_test_ready = preprocessor.transform(X_test)        # reuses what was learned
print(X_train_ready.shape, X_test_ready.shape)

Step 4 — attach a model

from sklearn.linear_model import LogisticRegression

model = Pipeline([
    ("prep", preprocessor),
    ("clf", LogisticRegression(max_iter=1000)),
])

model.fit(X_train, y_train)
print("Test accuracy:", model.score(X_test, y_test))

Why pipelines are better

Benefit	Explanation
No leakage	Statistics are learned only from the training data
Cross-validation safe	Every fold refits the preprocessing on its own training part
Reproducible	One object holds all steps and can be saved with joblib
Deployment-ready	New data goes through the same steps automatically