Data preparation is a sequence of steps. Each step in this notebook has its own chapter.
| Step | Goal | Chapter |
|---|---|---|
| 1 | Understand shape, types and quality of the data | Introduction |
| 2 | Remove duplicates, fix types and formats | Data Cleaning |
| 3 | Remove or fill blank values | Missing Values |
| 4 | Detect and treat extreme values | Outliers |
| 5 | Create train / validation / test sets | Splitting Data |
| 6 | Keep only the useful features | Feature Selection |
| 7 | Bring features to comparable ranges | Feature Scaling |
| 8 | Create new informative features, encode categories | Feature Engineering |
| 9 | Compress or transform features (PCA, TF-IDF) | Feature Extraction |
| 10 | Fix skewed class distributions | Imbalanced Data |
The order is flexible, but one rule is not
Anything that learns from the data — imputation values, outlier bounds, scaling statistics, selected features, PCA components, resampling — must be learned from the training set only and then applied to validation and test data.
This is why data splitting usually happens before imputation, scaling, selection and resampling.
A typical order in practice
- Inspect the data and clean obvious errors (duplicates, wrong types).
- Split into train and test sets.
- Inside a pipeline fitted on the training set: impute, treat outliers, encode, engineer, scale, select or extract features.
- Resample the training set if classes are imbalanced.
- Train and evaluate the model.