Interpolation
Interpolation estimates a missing value from the trend or pattern of the surrounding points. It is more informative than simple imputation, especially for ordered data.
linear_interpolation = df["Marks"].interpolate(method="linear")
quadratic_interpolation = df["Marks"].interpolate(method="quadratic")
print("Linear Interpolation:\n", linear_interpolation)
print("Quadratic Interpolation:\n", quadratic_interpolation)
- Linear interpolation assumes a straight line between two adjacent non-missing values.
- Quadratic interpolation assumes a quadratic curve that passes through three adjacent non-missing values (needs SciPy).
- For time series you can also use
method="time"on a datetime index.
Pros: captures trends and preserves relationships. Cons: assumes a pattern (linear, quadratic) and is meaningless for unordered data.
Model-based imputation
Predict the missing value from the other columns.
KNN Imputer
Fills a value using the average of the k most similar rows.
from sklearn.impute import KNNImputer
num = df[["Marks", "Rank"]]
knn = KNNImputer(n_neighbors=3)
num_knn = pd.DataFrame(knn.fit_transform(num), columns=num.columns)
print(num_knn)
Iterative Imputer
Models each column with missing values as a function of the other columns (similar to the MICE method).
from sklearn.experimental import enable_iterative_imputer # noqa: F401
from sklearn.impute import IterativeImputer
mice = IterativeImputer(random_state=0)
num_mice = pd.DataFrame(mice.fit_transform(num), columns=num.columns)
print(num_mice)
Pros: uses relationships between features, usually more accurate. Cons: slower, more complex, may overfit small datasets.
Missing-indicator features
Sometimes being missing is informative (MNAR). Keep that signal by adding a flag column.
df["Marks_missing"] = df["Marks"].isnull().astype(int)
# or let scikit-learn add the indicator columns
from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy="median", add_indicator=True)