Detecting Missing Data - Missing Values | Data Preparation

Before treating missing values you must first find them, including the ones hidden behind placeholders.

How missing values are represented

Blank cells in spreadsheets or CSV files.
Special values such as "NA", "NaN", "NULL", "?" or numbers like -999.
Codes or flags such as "MISSING" or "UNKNOWN".

Use one consistent, well-documented representation. Convert placeholders to real NaN first, otherwise pandas will not detect them.

df = pd.read_csv("data.csv", na_values=["NA", "N/A", "?", "-999", "Unknown"])

# or after loading
df = df.replace({-999: np.nan, "Unknown": np.nan})

Useful pandas functions

Function	Purpose
`isnull()` / `isna()`	Boolean mask of missing values
`notnull()`	Boolean mask of non-missing values
`info()`	Summary with non-null counts
`dropna()`	Remove rows or columns with missing values
`fillna()`	Fill missing values
`replace()`	Replace specific values
`drop_duplicates()`	Remove duplicate rows
`unique()`	Get unique values

Create a sample DataFrame

We will use this dataset for the rest of the chapter.

import pandas as pd
import numpy as np

data = {
    "School ID": [101, 102, 103, np.nan, 105, 106, 107, 108],
    "Name": ["A", "B", "C", "D", "E", "F", "G", "H"],
    "Address": ["123 Main St", "456 Oak Ave", "789 Pine Ln", "101 Elm St",
                np.nan, "222 Maple Rd", "444 Cedar Blvd", "555 Birch Dr"],
    "City": ["Mumbai", "Delhi", "Bengaluru", "Chennai", "Kolkata", np.nan, "Pune", "Jaipur"],
    "Subject": ["Math", "English", "Science", "Math", "History", "Math", "Science", "English"],
    "Marks": [85, 92, 78, 89, np.nan, 95, 80, 88],
    "Rank": [2, 1, 4, 3, 8, 1, 5, 3],
    "Grade": ["B", "A", "C", "B", "D", "A", "C", "B"],
}

df = pd.DataFrame(data)
print("Sample DataFrame:")
print(df)

Count and locate missing values

print(df.isnull().sum())                     # count per column
print((df.isnull().mean() * 100).round(1))   # percentage per column
print(df[df.isnull().any(axis=1)])           # rows that contain a missing value

In the sample data, School ID, Address, City and Marks each have one missing value.