Gradient Descent Explained: Learning Rate, SGD and Adam

Introduction

Training a machine-learning model means finding the parameters that make its predictions as good as possible. We measure "how bad" the model is with a loss function (also called a cost or objective function), so training becomes an optimisation problem: find the parameters that minimise the loss.

Gradient descent is the most widely used method for doing this. It is behind linear regression, logistic regression, neural networks, deep learning and large language models.

The hill-walking picture

Imagine you are standing on a foggy hillside and want to reach the lowest valley. You cannot see far, but you can feel which direction slopes downhill most steeply under your feet. So you take a small step that way, feel the slope again, and repeat. That is gradient descent.

The landscape is the loss function.
Your position is the current set of parameters.
The slope is the gradient.
The step size is the learning rate.

The algorithm

Let θ be the vector of parameters and L(θ) the loss.

Initialise θ (randomly or with zeros).
Compute the gradient ∇L(θ): the derivative of the loss with respect to every parameter.
Update the parameters by moving against the gradient:

θ  ←  θ − η · ∇L(θ)

Repeat steps 2 and 3 until the loss stops improving (convergence) or you reach a maximum number of iterations.

Here η (eta, sometimes α) is the learning rate: how big a step to take.

Why the minus sign? The gradient points uphill, so subtracting it moves downhill.

A one-dimensional example, step by step

Minimise f(x) = (x − 3)². The gradient is f′(x) = 2(x − 3), and the true minimum is at x = 3.

Start at x₀ = 0 with η = 0.1:

Step	x	Gradient 2(x − 3)	New x = x − 0.1 × gradient
0	0	−6	0.6
1	0.6	−4.8	1.08
2	1.08	−3.84	1.464
3	1.464	−3.072	1.7712
...	...	...	→ 3

Each step multiplies the remaining distance to the minimum by (1 − 2η) = 0.8, so x gets closer and closer to 3.

A two-dimensional example

Minimise f(x, y) = x² + y². The gradient is ∇f = [2x, 2y]. Start at (4, −2) with η = 0.25:

gradient = [8, −4]        →  new point = [4 − 0.25×8, −2 + 0.25×4] = [2, −1]
gradient = [4, −2]        →  new point = [1, −0.5]

The point halves each time, spiralling straight towards the minimum at (0, 0).

The learning rate matters

The learning rate is the most important hyperparameter of gradient descent.

Learning rate	Behaviour
Too small	Very slow convergence; may stall before reaching the minimum
Just right	Smooth, steady decrease of the loss
Too large	Overshoots the minimum, oscillates, or diverges (the loss explodes)

For f(x) = x², each step multiplies x by (1 − 2η):

η = 0.1: factor 0.8, converges smoothly.
η = 0.5: factor 0, converges in one step.
η = 1.0: factor −1, x bounces between +1 and −1 forever.
η = 1.1: factor −1.2, diverges (1 → −1.2 → 1.44 → ...).

Learning-rate schedules

Instead of one fixed rate, many training runs decay the learning rate over time (step decay, exponential decay, cosine annealing) or use a short warm-up at the start.

When to stop

The loss changes by less than a tiny tolerance.
The gradient norm ‖∇L‖ is near zero.
A maximum number of iterations (epochs) is reached.
The validation loss starts increasing (early stopping), a sign of overfitting.

Gradient descent for linear regression

Model: ŷ = w·x + b. Loss: mean squared error over m samples.

L(w, b) = (1/m) Σ (ŷᵢ − yᵢ)²

The gradients are:

∂L/∂w = (2/m) Σ (ŷᵢ − yᵢ) · xᵢ
∂L/∂b = (2/m) Σ (ŷᵢ − yᵢ)

and each update is w ← w − η·∂L/∂w and b ← b − η·∂L/∂b.

One step by hand. Data: (1, 2), (2, 4), (3, 6) (so y = 2x). Start with w = 0, b = 0 and η = 0.1.

Predictions ŷ = 0, so errors (ŷ − y) = −2, −4, −6.
∂L/∂w = (2/3)(−2·1 − 4·2 − 6·3) = (2/3)(−28) ≈ −18.67
∂L/∂b = (2/3)(−2 − 4 − 6) = (2/3)(−12) = −8
w = 0 − 0.1 × (−18.67) = 1.867, b = 0 − 0.1 × (−8) = 0.8

After one step w has jumped from 0 to 1.867, already close to the true value 2.

Gradient descent for logistic regression

For binary classification with p = σ(w·x + b) and log-loss, the gradient has the beautifully simple form:

∂L/∂w = (p − y) · x
∂L/∂b = (p − y)

Variants of gradient descent

Variant	Data used per update	Pros	Cons
Batch GD	The entire dataset	Stable, smooth path	Slow and memory-hungry for big data
Stochastic GD (SGD)	One sample	Very fast updates; noise can help escape poor regions	Very noisy path
Mini-batch GD	A small batch (32 to 512 samples)	Good balance; uses GPU efficiently	Needs a batch size to be chosen

Mini-batch gradient descent is the standard choice in deep learning. One pass through the whole dataset is an epoch. With 10,000 samples and batch size 100, one epoch consists of 100 updates.

Smarter optimisers

Momentum

Keeps a running "velocity" so updates build speed along consistent directions and damp oscillations:

v ← β·v + ∇L(θ)
θ ← θ − η·v          (β is typically 0.9)

RMSProp

Divides each step by a running average of recent squared gradients, so parameters with large gradients take smaller steps and vice versa.

Adam

Combines momentum and RMSProp with bias correction. It is a very popular default optimiser: Adam(lr=0.001) works well on a wide range of problems.

Challenges and pitfalls

Feature scaling. If one feature ranges over 0 to 1 and another over 0 to 100,000, the loss surface becomes a long narrow valley and gradient descent zig-zags. Standardise the features first.
Local minima and saddle points. Non-convex losses (neural networks) have flat regions and saddle points. Noise from mini-batches and momentum help.
Vanishing and exploding gradients. In deep networks the gradients can shrink to zero or blow up. Remedies: ReLU activations, careful initialisation, normalisation layers, residual connections and gradient clipping.
Choosing the learning rate. Try values on a log scale (0.1, 0.01, 0.001, ...) and plot the loss curve.
Convex versus non-convex. For a convex loss (linear and logistic regression) gradient descent finds the global minimum. For non-convex losses it usually finds a good solution, not necessarily the best.

Real-life applications

Training neural networks: image recognition, speech recognition, translation, chatbots.
Recommender systems: learning user and item factors.
Finance: fitting risk and pricing models.
Engineering and control: tuning parameters of physical systems.
Any optimisation problem where a derivative can be computed.

Where it appears in machine learning

Almost every modern model is trained by some variant of gradient descent: linear and logistic regression, support vector machines (in some forms), neural networks, gradient boosting (in function space), and large language models (Adam-style optimisers over billions of parameters).

Python implementation

Minimising a simple function

x = 0.0
learning_rate = 0.1

for step in range(50):
    grad = 2 * (x - 3)
    x = x - learning_rate * grad

print(x)      # very close to 3

Linear regression from scratch

import numpy as np

rng = np.random.default_rng(0)
X = rng.uniform(0, 10, size=100)
y = 3 * X + 4 + rng.normal(0, 1, size=100)      # true w = 3, b = 4

w, b = 0.0, 0.0
lr = 0.01
m = len(X)
losses = []

for epoch in range(2000):
    y_pred = w * X + b
    error = y_pred - y

    losses.append(np.mean(error ** 2))

    dw = (2 / m) * np.sum(error * X)
    db = (2 / m) * np.sum(error)

    w -= lr * dw
    b -= lr * db

print(w, b)             # close to 3 and 4
print(losses[0], losses[-1])

Mini-batch SGD with scikit-learn

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

model = make_pipeline(
    StandardScaler(),
    SGDRegressor(learning_rate="constant", eta0=0.01, max_iter=1000, random_state=0),
)
model.fit(X.reshape(-1, 1), y)
print(model.predict([[5.0]]))       # close to 3 * 5 + 4 = 19

The same idea in PyTorch

import torch

X_t = torch.tensor(X, dtype=torch.float32).unsqueeze(1)
y_t = torch.tensor(y, dtype=torch.float32).unsqueeze(1)

model = torch.nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = torch.nn.MSELoss()

for epoch in range(2000):
    optimizer.zero_grad()
    loss = loss_fn(model(X_t), y_t)
    loss.backward()           # compute gradients (automatic differentiation)
    optimizer.step()          # update the parameters

print(model.weight.item(), model.bias.item())

Solved examples

Example 1: One-dimensional descent

Minimise f(x) = (x − 3)² starting at x = 0 with η = 0.1. Find the first four iterates.

Solution. The update is x ← x − 0.1 × 2(x − 3) = 0.8x + 0.6.

x₁ = 0.8(0) + 0.6 = 0.6
x₂ = 0.8(0.6) + 0.6 = 1.08
x₃ = 0.8(1.08) + 0.6 = 1.464
x₄ = 0.8(1.464) + 0.6 = 1.7712

The values approach the minimum x = 3.

Example 2: Two variables

Minimise f(x, y) = x² + y² from (4, −2) with η = 0.25. Find the next two points.

Solution. ∇f = [2x, 2y].

At (4, −2): gradient = [8, −4], so the new point is (4 − 2, −2 + 1) = (2, −1).
At (2, −1): gradient = [4, −2], so the new point is (2 − 1, −1 + 0.5) = (1, −0.5).

Example 3: Effect of the learning rate

For f(x) = x² starting at x = 1, compare η = 0.1, η = 1.0 and η = 1.1.

Solution. The update is x ← x − η·2x = (1 − 2η)x.

η = 0.1: factor 0.8, so x = 0.8, 0.64, ... (converges).
η = 1.0: factor −1, so x = −1, 1, −1, ... (oscillates forever).
η = 1.1: factor −1.2, so x = −1.2, 1.44, −1.728, ... (diverges).

Example 4: One step of linear regression

Using data (1, 2), (2, 4), (3, 6), w = 0, b = 0 and η = 0.1, perform one gradient-descent step.

Solution. Errors = [−2, −4, −6].

∂L/∂w = (2/3)(−2 − 8 − 18) = −18.67; ∂L/∂b = (2/3)(−12) = −8.

New w = 0 + 1.867 = 1.867, new b = 0 + 0.8 = 0.8.

Example 5: Number of updates per epoch

A dataset has 10,000 samples. How many parameter updates occur in one epoch with (a) batch GD, (b) SGD, (c) mini-batches of 100?

Solution. (a) 1, (b) 10,000, (c) 10,000/100 = 100.

Example 6: Momentum

With β = 0.9 and a constant gradient of 1 at each step, compute the velocity for the first three steps (v starts at 0).

Solution. v₁ = 0.9(0) + 1 = 1, v₂ = 0.9(1) + 1 = 1.9, v₃ = 0.9(1.9) + 1 = 2.71. The velocity builds up, so consistent directions produce larger steps.

Example 7: Logistic-regression gradient

One sample has x = 2 and y = 1. The current parameters are w = 0 and b = 0. Find the gradients.

Solution. p = σ(0) = 0.5. ∂L/∂w = (p − y)·x = (0.5 − 1)(2) = −1. ∂L/∂b = p − y = −0.5. Both are negative, so w and b should increase to raise the predicted probability of class 1.

Common mistakes

Using a learning rate that is too large and watching the loss explode (or too small and concluding "the model does not learn").
Forgetting to scale features.
Adding the gradient instead of subtracting it.
Not shuffling the data between epochs when using SGD.
Stopping only when the training loss stops falling instead of monitoring the validation loss.
In PyTorch, forgetting optimizer.zero_grad(), so gradients accumulate across steps.

Practice problems

Minimise f(x) = x² − 4x from x₀ = 0 with η = 0.25. Compute x₁ and x₂. (The true minimum is at x = 2.)
For f(x, y) = x² + 3y², compute ∇f at (2, 1) and one update with η = 0.1.
A dataset has 2,000 samples. How many iterations are there per epoch with mini-batches of 50?
What happens if the learning rate is far too large?
For f(x) = x², what is the new x after one step from x = 3 with η = 0.5?
With β = 0.5 and gradients 2, 2 in two consecutive steps, compute the momentum velocity v₂ (start with v = 0).
For the model ŷ = w·x with a single data point (x = 2, y = 5), w = 1, and loss (ŷ − y)², compute the gradient and the updated w for η = 0.05.

Answers

f′(x) = 2x − 4. x₁ = 0 − 0.25(−4) = 1; x₂ = 1 − 0.25(−2) = 1.5
∇f = [2x, 6y] = [4, 6]; new point = (2 − 0.4, 1 − 0.6) = (1.6, 0.4)
2000/50 = 40 iterations
The steps overshoot the minimum; the loss oscillates or diverges to infinity
Gradient = 6; x = 3 − 0.5×6 = 0 (the minimum, reached in one step)
v₁ = 2; v₂ = 0.5×2 + 2 = 3
Error = 2 − 5 = −3; gradient = 2 × 2 × (−3) = −12; w = 1 − 0.05×(−12) = 1.6

Key takeaways

Gradient descent minimises a loss by repeatedly stepping against the gradient: θ ← θ − η∇L(θ).
The learning rate controls the step size: too small is slow, too large diverges.
Batch, stochastic and mini-batch versions trade off stability and speed; mini-batch is the deep-learning standard.
Momentum, RMSProp and Adam speed up and stabilise training.
Scale your features, monitor the loss curve, and use early stopping.
Almost every modern ML model, from linear regression to large neural networks, is trained with a form of gradient descent.