Now that we know the problems, we fix them.
Step 6 — Drop irrelevant or mostly-missing columns
df.drop(columns=[...])removes columns.df.dropna(subset=[...])removes rows where the listed columns are missing.fillna()fills gaps with a value such as the mean or median.
df1 = df.drop(columns=["Name", "Ticket", "Cabin"])
df1 = df1.dropna(subset=["Embarked"]) # only 2 rows missing
df1["Age"] = df1["Age"].fillna(df1["Age"].mean()) # median is often safer
Step 7 — Fix structural errors
Strip whitespace, standardize case and correct data types.
df1["Sex"] = df1["Sex"].str.strip().str.lower()
df1["Embarked"] = df1["Embarked"].str.strip().str.upper()
df1["Pclass"] = df1["Pclass"].astype("category")
Step 8 — Detect outliers with a box plot
plt.boxplot() shows the median, quartiles and points beyond the whiskers, which are potential outliers.
import matplotlib.pyplot as plt
plt.boxplot(df1["Age"], vert=False)
plt.xlabel("Age")
plt.title("Box Plot")
plt.show()
Step 9 — Calculate outlier boundaries and remove them
One simple rule: values outside mean ± 2 x standard deviation are treated as outliers.
mean = df1["Age"].mean()
std = df1["Age"].std()
lower_bound = mean - 2 * std
upper_bound = mean + 2 * std
print("Lower bound:", round(lower_bound, 2), "| Upper bound:", round(upper_bound, 2))
df2 = df1[(df1["Age"] >= lower_bound) & (df1["Age"] <= upper_bound)]
print("Rows removed:", len(df1) - len(df2))
Repeating this rule again and again keeps removing more rows, because the mean and standard deviation change every time. Apply it once, or prefer the IQR rule and domain knowledge (see the Outliers chapter). An age of 65 or 70 is rare but perfectly valid on a ship.
Step 10 — Validate the result
Re-run the quality checks to confirm that the problems are gone.
print(df2.isnull().sum()) # missing values per column
print(df2.duplicated().sum()) # duplicate rows
print(df2.dtypes) # data types
Expected result: no missing values in the columns we used, no duplicate rows and correct data types.