Feature engineering is the process of creating, transforming and selecting input variables so that machine learning algorithms can learn patterns more effectively. Raw columns rarely express everything that matters; a good data scientist adds domain knowledge to turn raw data into informative features.
Well-engineered features often improve a model more than switching to a fancier algorithm.
Why is it important?
- Models can only learn from the information present in the features.
- A good feature can make a hard problem almost linear.
- It can reduce the need for complex models and large amounts of data.
- It makes models easier to explain to non-technical people.
A simple example
Predicting house prices from length and width: the model may struggle, but a new feature area = length x width is directly related to price.
import pandas as pd
houses = pd.DataFrame({"length": [10, 12, 8], "width": [8, 9, 7]})
houses["area"] = houses["length"] * houses["width"] # engineered feature
print(houses)
Main steps of feature engineering
| Step | Description | Example |
|---|---|---|
| Feature creation | Build new features from existing ones | FamilySize = SibSp + Parch + 1 |
| Feature transformation | Change scale, shape or distribution | Log transform, binning |
| Feature encoding | Convert categories or text into numbers | One-hot encoding |
| Feature scaling | Bring features to comparable ranges | Standardization |
| Feature selection | Keep the most useful features | Lasso, RFE |
| Feature extraction | Compress into new representations | PCA, TF-IDF |
Scaling, selection and extraction have their own chapters. In this chapter we focus on creation, transformation and encoding.
The feature engineering workflow
- Understand the problem and the data.
- Brainstorm features using domain knowledge.
- Create the features.
- Check whether they help (cross-validation).
- Keep what works, drop what does not, and repeat.