Advanced EDA: Overview

Basic EDA answers "what does my data look like?". Advanced EDA goes further: it studies the structure and characteristics of the dataset so that the modelling stage starts on solid ground.

It helps to:

• Evaluate data quality by finding missing values and inconsistencies,
• Select useful variables for model development,
• Support decisions in preprocessing and model design.

The advanced EDA toolbox

Area	What it does	Typical tools
Descriptive statistics	Summarise centre, spread and shape	mean, median, mode, std, IQR, skewness, kurtosis
Distribution visualisation	See the shape of data and group differences	bar, stacked bar, histogram, box, violin, pie, heatmap, scatter
Feature interactions	Compare across categories, view many pairs	facet grids, pair plots
Outlier detection	Find anomalous points	Z-score, IQR, Isolation Forest, LOF
Feature engineering	Create better inputs	log transform, polynomial, interaction features
Dimensionality reduction	Simplify many features	PCA, t-SNE, UMAP

Where it fits in a project

Raw data -> Cleaning -> Basic EDA -> ADVANCED EDA -> Feature engineering -> Modelling -> Evaluation
                                          ^                                        |
                                          +----------- return when model surprises you

A practical checklist

1. Summarise every numeric column (mean, median, std, skewness, kurtosis).
2. Plot each distribution; mark strong skew and multimodality.
3. Detect outliers with at least two methods; decide keep / cap / remove.
4. Check target balance and relationships with every feature.
5. Look at the correlation matrix and remove redundancy.
6. Look at interactions across categories with facet grids.
7. Engineer features (log, polynomial, interactions, binning).
8. Visualise structure with PCA / t-SNE / UMAP.
9. Write a short list of decisions and assumptions.

A one-cell "automated" starting report

def quick_report(df):
    out = pd.DataFrame({
        "dtype": df.dtypes,
        "missing_%": (df.isnull().mean() * 100).round(2),
        "unique": df.nunique(),
    })
    num = df.select_dtypes("number")
    out.loc[num.columns, "skew"] = num.skew().round(2)
    out.loc[num.columns, "kurtosis"] = num.kurt().round(2)
    return out

print(quick_report(df))

The remaining lessons of this chapter walk through each area in detail.