Handling Missing and Duplicate Values

Clean data is the foundation of reliable analysis. Two of the most common problems are missing values and duplicate rows.

# Missing values per column
print(df.isnull().sum())

# Percentage of missing values
print((df.isnull().mean() * 100).round(2))

# Duplicate rows
print(df.duplicated().sum())

# Typical fixes
df["alcohol"] = df["alcohol"].fillna(df["alcohol"].median())  # impute numeric
df = df.dropna(subset=["quality"])                             # drop if target missing
df = df.drop_duplicates().reset_index(drop=True)               # remove duplicates

Checking missing values

df.isnull() returns a True/False table; .sum() adds up the True values per column. Looking at the percentage is even better: 2% missing is easy to fix, 60% missing may mean the column should be dropped.

Why are values missing?

Type	Meaning	Example
MCAR (Missing Completely At Random)	Missing for no reason	A sensor randomly failed
MAR (Missing At Random)	Depends on another observed column	Older people skip an online form field
MNAR (Missing Not At Random)	Depends on the missing value itself	High earners hide their income

The type matters: simple filling is safe for MCAR, risky for MNAR.

Strategies for missing values

Drop rows - df.dropna(). Fine when only a tiny fraction of rows is affected.
Drop the column - df.drop(columns=["col"]). Consider when most of it is missing.
Fill with mean - for roughly symmetric numeric data.
Fill with median - for skewed numeric data or data with outliers (safer default).
Fill with mode - for categorical columns.
Forward / backward fill - df.ffill() for time series.
Model based - KNNImputer, IterativeImputer from scikit-learn.
Add an indicator - df["col_missing"] = df["col"].isnull() because "being missing" itself can be informative.

Duplicate rows

df.duplicated().sum() counts rows that are exact copies of an earlier row. Duplicates make some observations count twice, which biases statistics and leaks data between training and test sets.

# See the duplicates before deleting them
dups = df[df.duplicated(keep=False)]
print(dups.sort_values(by=list(df.columns)).head())

# Duplicates on selected columns only
df.drop_duplicates(subset=["fixed acidity", "alcohol", "quality"])

Caution: in WineQT the Id column makes every row unique. If you suspect duplicated measurements, check duplicates after excluding Id:

print(df.drop(columns=["Id"]).duplicated().sum())

Mini checklist

Always check missing values and duplicates before plotting.
Never fill missing values before splitting train/test using statistics of the whole data (data leakage).
Record every cleaning decision so the analysis is reproducible.