First Look at a Dataset

Always inspect the data before changing anything. A few lines of pandas reveal most problems.

Load and inspect

import pandas as pd
import numpy as np

df = pd.read_csv("Titanic-Dataset.csv")

print(df.shape)               # (rows, columns)
print(df.head())              # first 5 rows
df.info()                     # dtypes and non-null counts
print(df.describe())          # summary statistics
print(df.isnull().sum())      # missing values per column
print(df.duplicated().sum())  # duplicate rows

Command	What it tells you
df.shape	Number of rows and columns
df.head()	A quick look at the first rows
df.info()	Column names, data types, non-null counts
df.describe()	Mean, std, min, max, quartiles of numeric columns
df.isnull().sum()	Missing values per column
df.duplicated().sum()	Number of duplicate rows
df.nunique()	Number of distinct values per column

Look at categories and the target

# categorical columns and their unique values
cat_cols = df.select_dtypes(include="object").columns
print(df[cat_cols].nunique())

# target distribution
print(df["Survived"].value_counts())
print(df["Survived"].value_counts(normalize=True).round(3))

Questions to answer at this stage

1. What does each column mean, and which one is the target?
2. Which columns are numeric, categorical, text or dates?
3. How much data is missing, and in which columns?
4. Are there impossible values (negative age, future dates)?
5. Is the target balanced or skewed?
6. Are there columns that should not be used (IDs, columns that leak the target)?