Feature Engineering: Log, Polynomial and Interaction Features

Feature engineering turns raw data into inputs that help a model learn. EDA tells you what to engineer: skew suggests a log transform, curves suggest polynomial terms, and interactions found in facet grids suggest product features.

from sklearn.preprocessing import PolynomialFeatures

# 1) Log transformation for right-skewed data
df["sulphates_log"] = np.log1p(df["sulphates"])
print(df["sulphates"].skew(), "->", df["sulphates_log"].skew())

# 2) Polynomial features (degree 2)
poly = PolynomialFeatures(degree=2, include_bias=False)
arr = poly.fit_transform(df[["alcohol", "density"]])
print(poly.get_feature_names_out(["alcohol", "density"]))
# ['alcohol' 'density' 'alcohol^2' 'alcohol density' 'density^2']

# 3) A hand-made interaction feature
df["alcohol_x_sulphates"] = df["alcohol"] * df["sulphates"]

1. Log transformation

Compresses large values, reducing right skew and the influence of extreme values.

• Best for: strongly right-skewed or exponentially growing data - income, population, prices, counts.
• Not suitable: data that is already roughly normal (it would distort it), or data containing zero / negative values without adjustment. Use np.log1p(x) (log of 1 + x) for zeros; use Box-Cox or Yeo-Johnson for more general cases.
• Example: household incomes are highly skewed with a few extreme earners; log(income) is far more symmetric and helps linear models.

from sklearn.preprocessing import PowerTransformer
pt = PowerTransformer(method="yeo-johnson")     # handles zero and negative values
df[["chlorides_t"]] = pt.fit_transform(df[["chlorides"]])

Check it worked: compare skewness before and after, and re-plot the histogram.

2. Polynomial features

Create squares, cubes and cross-terms of existing features so that linear models can capture curved relationships.

• Best for: non-linear relationship between a feature and the target, e.g. house price versus square footage that flattens for very large houses.
• Not suitable: truly linear relationships, or high degrees - they cause overfitting and huge feature counts. Rarely go beyond degree 2 or 3, and use regularisation (Ridge / Lasso).
• Example: adding sqft^2 lets a linear regression bend.

For two features a and b with degree 2, the new columns are a, b, a^2, a*b, b^2.

3. Interaction features

Combine two or more features so the model can represent effects that depend on each other.

• Best for: when you believe the impact of one feature depends on another - e.g. age and income jointly influence luxury purchases.
• Not suitable: adding many combinations without justification (overfitting, hard to interpret). Add only meaningful, explainable interactions.
• Example: a retailer creates age x income. Young high earners behave differently from older high earners, and the product term captures that nuance.

# Ratio and difference features are interactions too
df["free_sulfur_ratio"] = df["free sulfur dioxide"] / df["total sulfur dioxide"]
df["acid_balance"] = df["fixed acidity"] - df["volatile acidity"]

Other common engineering steps

Technique	Use
Binning (pd.cut, pd.qcut)	turn numbers into groups such as Low / Medium / High
Scaling (StandardScaler, MinMaxScaler)	put features on a similar scale for distance / gradient methods
One-hot / ordinal encoding	make categorical data numeric
Date parts	extract year, month, weekday, is_weekend from timestamps
Aggregations	customer's average order value, count of purchases
Text features	length, word counts, TF-IDF

Pipeline tip: avoid data leakage

Fit transformers on the training set only, then apply to the test set:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split, cross_val_score

X = df[["alcohol", "sulphates", "volatile acidity"]]
y = df["quality"]

pipe = Pipeline([
    ("poly", PolynomialFeatures(degree=2, include_bias=False)),
    ("scale", StandardScaler()),
    ("model", Ridge(alpha=1.0)),
])
print(cross_val_score(pipe, X, y, cv=5, scoring="r2").mean())

Measure every new feature by whether it improves validation performance, not merely whether it looks clever.