Let's apply the cleaning steps to a real dataset: the Titanic passenger dataset (Titanic-Dataset.csv, available from Kaggle). We start by inspecting the data.
Step 1 — Import libraries and load the dataset
import pandas as pd
import numpy as np
df = pd.read_csv("Titanic-Dataset.csv")
df.info()
df.head()
df.info() shows 12 columns, their types and how many values are non-null. df.head() shows the first rows.
Step 2 — Check for duplicate rows
df.duplicated() returns a boolean Series that is True for each repeated row.
print("Duplicate rows:", df.duplicated().sum())
df = df.drop_duplicates()
Step 3 — Identify column data types
object dtype usually means text or categorical data. A list comprehension over df.columns splits columns into categorical and numerical.
cat_col = [col for col in df.columns if df[col].dtype == "object"]
num_col = [col for col in df.columns if df[col].dtype != "object"]
print("Categorical columns:", cat_col)
print("Numerical columns:", num_col)
Step 4 — Count unique values in categorical columns
df[cat_col].nunique() returns the number of distinct values in each categorical column.
df[cat_col].nunique()
Observation: Name and Ticket have a unique or almost-unique value per passenger, so they are poor direct features. Sex and Embarked have only 2-3 values and are true categorical variables.
Step 5 — Missing values as a percentage
df.isnull() returns a boolean DataFrame. Summing per column and dividing by the number of rows gives the share of missing data.
missing_pct = round((df.isnull().sum() / df.shape[0]) * 100, 2)
print(missing_pct.sort_values(ascending=False))
Typical findings:
| Column | Missing | Decision |
|---|---|---|
Cabin | very high (~77%) | Drop, or convert to a HasCabin flag |
Age | ~20% | Impute (median or model-based) |
Embarked | 2 rows only | Drop those rows or fill with the mode |