Differentiation, Gradients and the Chain Rule for ML

Introduction

Differentiation is the part of calculus that measures how fast something changes. The result of differentiating a function is called its derivative.

If a car's position changes with time, its derivative is the velocity.
If a model's loss changes as a weight changes, the derivative tells us how much the loss will go up or down if we nudge that weight.

That last idea is the entire foundation of how machine learning models are trained. To reduce a model's error we need to know which direction to adjust each parameter, and derivatives supply exactly that information.

The idea: slope of a curve

For a straight line, the slope is constant: "rise over run". For a curve, the slope changes from point to point. The derivative at a point is the slope of the tangent line to the curve at that point.

Positive derivative: the function is increasing there.
Negative derivative: the function is decreasing.
Zero derivative: the curve is momentarily flat (a possible minimum, maximum or saddle point).

Definition (limit form)

The derivative of f at x is the limit of the slope of a very short secant line:

f′(x) = lim (h → 0)  [ f(x + h) − f(x) ] / h

Example: derivative of f(x) = x².

[ (x + h)² − x² ] / h = (2xh + h²) / h = 2x + h    →   2x   as h → 0

So f′(x) = 2x. At x = 3, the slope is 6.

Notation

You will see several equivalent notations: f′(x), df/dx, dy/dx and Df.

Rules of differentiation

Rule	Formula
Constant	d/dx [c] = 0
Power	d/dx [xⁿ] = n·xⁿ⁻¹
Constant multiple	d/dx [c·f(x)] = c·f′(x)
Sum / difference	(f ± g)′ = f′ ± g′
Product	(f · g)′ = f′·g + f·g′
Quotient	(f / g)′ = (f′·g − f·g′) / g²
Chain rule	d/dx [f(g(x))] = f′(g(x)) · g′(x)

Derivatives of common functions

Function f(x)	Derivative f′(x)
xⁿ	n·xⁿ⁻¹
eˣ	eˣ
aˣ	aˣ · ln a
ln x	1/x
sin x	cos x
cos x	−sin x
Sigmoid σ(x) = 1/(1 + e⁻ˣ)	σ(x)(1 − σ(x))
tanh x	1 − tanh²(x)
ReLU max(0, x)	0 for x < 0, and 1 for x > 0

The chain rule: the engine of backpropagation

When one function is nested inside another, y = f(g(x)), the derivative is the product of the derivatives of each layer:

dy/dx = (dy/dg) · (dg/dx)

Example. For y = (3x + 2)⁵, let g = 3x + 2. Then dy/dg = 5g⁴ and dg/dx = 3, so dy/dx = 15(3x + 2)⁴.

A neural network is a long chain of nested functions (layer after layer). Backpropagation is simply the chain rule applied efficiently from the output back to every weight.

Higher-order derivatives and optimisation

The second derivative f″(x) measures how the slope itself changes (the curvature).

To find where a function has a minimum or maximum:

Solve f′(x) = 0 to find critical points.
Check the second derivative:
- f″(x) > 0: the curve bends upward, so it is a local minimum.
- f″(x) < 0: it bends downward, so it is a local maximum.
- f″(x) = 0: inconclusive (could be a saddle or inflection point).

A function whose second derivative is always positive is convex and has a single global minimum. Squared-error loss for linear regression is convex, which is why it is so well-behaved.

Functions of several variables

Real models have many parameters: a loss L(w₁, w₂, ..., wₙ). We need derivatives with respect to each one.

Partial derivatives

A partial derivative ∂f/∂x measures how f changes when only x changes, treating all other variables as constants.

Example. f(x, y) = x²y + 3y².

∂f/∂x = 2xy            (treat y as a constant)
∂f/∂y = x² + 6y        (treat x as a constant)

The gradient

The gradient collects all partial derivatives into a vector:

∇f = [ ∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ ]

Key facts:

The gradient points in the direction of steepest increase of f.
The negative gradient −∇f points in the direction of steepest decrease.
Where ∇f = 0, the function has a critical point.
The length ‖∇f‖ tells how steep the surface is.

Example. For f(x, y) = x²y + 3y² at the point (2, 1): ∂f/∂x = 2·2·1 = 4 and ∂f/∂y = 4 + 6 = 10, so ∇f(2, 1) = [4, 10].

Jacobian and Hessian (a preview)

The Jacobian is the matrix of first derivatives of a vector-valued function.
The Hessian is the matrix of second derivatives; it describes the curvature in all directions and is used by second-order optimisers.

Derivatives of common loss functions

For a single training example with prediction ŷ = w·x and target y:

Loss	Formula	Derivative with respect to ŷ
Squared error	(ŷ − y)²	2(ŷ − y)
Absolute error	\|ŷ − y\|	sign(ŷ − y)
Binary cross-entropy	−[y ln p + (1 − y) ln(1 − p)]	(p − y) / (p(1 − p)), which simplifies to (p − y) with respect to the logit z when p = σ(z)

By the chain rule, ∂L/∂w = (∂L/∂ŷ) · (∂ŷ/∂w). For squared error with ŷ = w·x: ∂L/∂w = 2x(w·x − y).

A tiny backpropagation example

Consider one neuron: z = w·x + b, prediction p = σ(z), loss L = (p − t)².

Take w = 0.5, b = 0, x = 2, target t = 1.

Forward pass: z = 1, p = σ(1) ≈ 0.7311, L ≈ 0.0723.

Backward pass (chain rule):

∂L/∂p = 2(p − t)      = 2(0.7311 − 1)   ≈ −0.5379
∂p/∂z = p(1 − p)      = 0.7311 × 0.2689 ≈  0.1966
∂z/∂w = x             = 2
∂L/∂w = (−0.5379) × 0.1966 × 2         ≈ −0.2115

The gradient is negative, so increasing w reduces the loss. A gradient-descent step would increase w. Frameworks such as PyTorch and TensorFlow do all this bookkeeping automatically (automatic differentiation).

Numerical differentiation and gradient checking

You can approximate a derivative with a tiny step h using the central difference:

f′(x) ≈ [ f(x + h) − f(x − h) ] / (2h)

This is not efficient for training, but it is a great way to check that your hand-derived gradients are correct (a "gradient check").

Real-life applications

Physics: velocity and acceleration are the first and second derivatives of position.
Economics: marginal cost and marginal revenue are derivatives.
Engineering: rates of heat flow, fluid flow and change in signals.
Medicine: the rate at which a drug concentration changes in the blood.
Optimisation: maximising profit or minimising cost or material.

Where it appears in machine learning

Training any model means minimising a loss, which needs the derivatives of the loss with respect to all parameters.
Backpropagation is the chain rule applied to a neural network.
Activation functions are chosen partly for having useful derivatives. Sigmoid and tanh derivatives shrink for large inputs, causing the vanishing gradient problem; ReLU keeps a derivative of 1 for positive inputs.
Optimisers (SGD, Adam) all consume gradients.
Sensitivity analysis and explainability: gradients show which input features most affect a prediction.

Python implementation

Numerical derivative

def f(x):
    return 3 * x**2 + 2 * x + 1

def numerical_derivative(func, x, h=1e-5):
    return (func(x + h) - func(x - h)) / (2 * h)

print(numerical_derivative(f, 1.0))     # about 8.0 (exact: 6x + 2 = 8)

Symbolic differentiation with SymPy

import sympy as sp

x, y = sp.symbols("x y")

f = 4*x**3 - 5*x**2 + 2*x - 7
print(sp.diff(f, x))                    # 12*x**2 - 10*x + 2

g = x**2 * y + 3 * y**2
print(sp.diff(g, x), sp.diff(g, y))     # 2*x*y   x**2 + 6*y
print(sp.diff(sp.exp(x) * x**2, x))     # x**2*exp(x) + 2*x*exp(x)

Automatic differentiation with PyTorch

import torch

w = torch.tensor(0.5, requires_grad=True)
x, b, t = 2.0, 0.0, 1.0

p = torch.sigmoid(w * x + b)
loss = (p - t) ** 2
loss.backward()

print(w.grad)       # tensor(-0.2115)

Solved examples

Example 1: From the definition

Find the derivative of f(x) = x² using the limit definition.

Solution. [(x + h)² − x²]/h = (2xh + h²)/h = 2x + h. As h → 0, this tends to 2x.

Example 2: Power rule

Differentiate f(x) = 4x³ − 5x² + 2x − 7.

Solution. f′(x) = 12x² − 10x + 2 (the constant −7 disappears).

Example 3: Product rule

Differentiate f(x) = x²·eˣ.

Solution. With u = x² and v = eˣ: f′ = 2x·eˣ + x²·eˣ = eˣ(x² + 2x).

Example 4: Chain rule

Differentiate f(x) = (3x + 2)⁵.

Solution. f′(x) = 5(3x + 2)⁴ · 3 = 15(3x + 2)⁴.

Example 5: Quotient rule

Differentiate f(x) = x/(x + 1).

Solution. f′ = [1·(x + 1) − x·1]/(x + 1)² = 1/(x + 1)².

Example 6: Finding a minimum

Find the minimum of f(x) = x² − 6x + 5.

Solution. f′(x) = 2x − 6 = 0, so x = 3. f″(x) = 2 > 0, so it is a minimum. The minimum value is f(3) = 9 − 18 + 5 = −4.

Example 7: Partial derivatives and the gradient

For f(x, y) = x²y + 3y², find ∇f at (2, 1).

Solution. ∂f/∂x = 2xy = 4 and ∂f/∂y = x² + 6y = 4 + 6 = 10, so ∇f(2, 1) = [4, 10].

Example 8: Derivative of the sigmoid

Find σ′(0).

Solution. σ(0) = 1/(1 + 1) = 0.5. σ′(0) = σ(0)(1 − σ(0)) = 0.5 × 0.5 = 0.25. This is the largest value the sigmoid derivative ever takes; for large |x| it approaches 0.

Example 9: Gradient of a squared-error loss

For one data point x = 3, y = 6 and a model ŷ = w·x, compute dL/dw at w = 1 with L = (wx − y)².

Solution. dL/dw = 2x(wx − y) = 2·3·(3 − 6) = −18. The gradient is negative, so increasing w will lower the loss (the correct weight is 2).

Common mistakes

Forgetting the chain rule (the "inner derivative") when differentiating a composite function.
Applying the power rule to eˣ (its derivative is eˣ, not x·eˣ⁻¹).
Treating other variables as changing when taking a partial derivative.
Assuming f′(x) = 0 always means a minimum. It could be a maximum or a saddle point.
Moving in the direction of the gradient (which increases the function) when you want to minimise.

Practice problems

Differentiate f(x) = 5x⁴ − 3x + 2.
Differentiate f(x) = x³ ln x.
Differentiate f(x) = e²ˣ.
Find the minimum of f(x) = x² + 4x + 1.
Find ∇f at (1, 2) for f(x, y) = 3x² + xy.
What is the derivative of ReLU at x = −3 and at x = 4?
Find the second derivative of x⁴.

Answers

20x³ − 3
Product rule: 3x²·ln x + x³·(1/x) = 3x² ln x + x²
Chain rule: 2e²ˣ
f′ = 2x + 4 = 0, so x = −2, and f(−2) = 4 − 8 + 1 = −3
∂f/∂x = 6x + y = 8 and ∂f/∂y = x = 1, so [8, 1]
0 at x = −3 and 1 at x = 4
First derivative 4x³, second derivative 12x²

Key takeaways

The derivative is the instantaneous rate of change, or the slope of the tangent line.
Learn the basic rules: power, sum, product, quotient and especially the chain rule.
For many variables, take partial derivatives; the vector of them is the gradient, which points in the direction of steepest increase.
Setting the derivative to zero finds critical points; the second derivative tells you which kind.
Backpropagation is the chain rule; training every ML model depends on derivatives.