The Data Cleaning Process

Data cleaning follows six broad steps.

1. Assess data quality

Check the dataset for:

• Missing values — blanks or nulls caused by incomplete collection, entry errors or data loss.
• Incorrect values — values outside the expected range or inconsistent with the data type.
• Format inconsistencies — the same kind of data written in different ways.

2. Remove irrelevant data

• Identify duplicate entries (by sorting, grouping or hashing) and remove them.
• Detect redundant observations that add no new information.
• Drop columns that are irrelevant to the analysis, such as unique IDs.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Alice", "Bob", "Carol", "David", "Eva", "Alice", None],
    "Date": ["2026-01-05", "2026-01-06", "2026-01-07", "2026-01-08",
             "2026-01-09", "2026-01-05", "2026-01-10"],
    "Score": [82, 75, 90, 68, 77, 82, 100],
})

print(df.duplicated())            # True for the repeated row
df = df.drop_duplicates()         # keep the first occurrence
df["Name"] = df["Name"].fillna("Unknown")
df["Date"] = pd.to_datetime(df["Date"], format="%Y-%m-%d")
print(df)

3. Fix structural errors

Structural errors are inconsistent formats, naming conventions or variable types.

• Standardize dates, times and other formats.
• Correct naming inconsistencies in column names and labels.
• Use the same units and scales everywhere (kg vs lb, 0-5 vs 0-10 ratings).

4. Handle missing data

• Impute with the mean, median or mode.
• Remove records when too much is missing or imputation is not possible.
• Use advanced imputation such as regression, k-nearest neighbors or decision trees.

In our student table, the missing Name in row 7 is filled with "Unknown" so the dataset stays complete and consistent.

5. Normalize data (organize the structure)

Data normalization in the database sense means splitting data into multiple tables where each table stores one type of information, reducing redundancy and keeping data consistent for efficient querying.

6. Identify and manage outliers

• Remove outliers that come from errors or are not representative of the population.
• Transform extreme but valid outliers (log, capping) to reduce their impact.

After each step, re-check the data. Cleaning is finished only when the quality checks from step 1 pass.