Step 11 — Data validation and verification
Validation means confirming that the data is accurate and consistent, for example by comparing it with external sources or expert knowledge. Here we also separate the independent features from the target.
- Independent features:
Pclass,Sex,Age,SibSp,Parch,Fare,Embarked. - Target:
Survived. PassengerIdis dropped because it does not affect survival.
X = df2[["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]]
Y = df2["Survived"]
Step 12 — Data formatting (scaling)
Formatting converts data into a standard structure that algorithms can process. The two most common techniques are:
- Min-Max scaling — rescales values to a range such as 0 to 1:
X_scaled = (X - X_min) / (X_max - X_min). - Standardization (Z-score) — gives mean 0 and standard deviation 1:
Z = (X - mean) / std.
from sklearn.preprocessing import MinMaxScaler, StandardScaler
num_cols = ["Age", "SibSp", "Parch", "Fare"]
# Min-Max scaling to [0, 1]
mm = MinMaxScaler(feature_range=(0, 1))
X_minmax = X.copy()
X_minmax[num_cols] = mm.fit_transform(X[num_cols])
# Standardization (Z-score)
ss = StandardScaler()
X_std = X.copy()
X_std[num_cols] = ss.fit_transform(X[num_cols])
print(X_minmax.head())
print(X_std.head())
Important: in real projects, split into train and test first, then fit the scaler on the training set only. Scaling the full dataset leaks test information (see the Splitting Data and Feature Scaling chapters).
Data cleaning strategies
- Understand the data — know its source, structure and domain.
- Document the process — record decisions, assumptions and rules.
- Prioritize critical issues — fix problems with the biggest impact first.
- Automate — use scripts or tools for repetitive tasks.
- Collaborate with domain experts — confirm the cleaned data meets business needs.
- Monitor and maintain — keep tracking data quality over time.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| Removing errors and irrelevant data helps the model learn better | Time-consuming for large, complex datasets |
| Data becomes accurate and consistent | Can lose important information if done carelessly |
| Data better represents underlying patterns | Needs time, effort, expertise and sometimes special tools |
| More reliable data quality | Removing too much data can cause underfitting |
| Helps find and remove sensitive information, improving security |