Matrices and Matrix Arithmetic for Machine Learning

Introduction

A matrix is a rectangular grid of numbers arranged in rows and columns. If vectors are the "nouns" of machine learning, matrices are the "verbs": they transform vectors, store whole datasets, and connect the layers of a neural network.

A matrix with m rows and n columns is called an m × n matrix (read "m by n"). The entry in row i and column j is written aᵢⱼ.

      ⎡ 1  2  3 ⎤
A  =  ⎢         ⎥         a 2 × 3 matrix, with a₁₂ = 2 and a₂₃ = 6
      ⎣ 4  5  6 ⎦

Matrices in machine learning

A dataset is stored as a data matrix X, where each row is one sample and each column is one feature:

              height  weight  age
sample 1  ⎡   1.70     65     29  ⎤
sample 2  ⎢   1.82     80     35  ⎥        X has shape (3 × 3)
sample 3  ⎣   1.60     55     22  ⎦

An image is a matrix of pixel intensities. A neural-network layer is a weight matrix. A covariance table is a matrix.

Types of matrices

Type	Description
Square	Same number of rows and columns (n × n)
Zero matrix	All entries 0
Identity matrix (I)	1s on the main diagonal, 0s elsewhere
Diagonal	Non-zero entries only on the main diagonal
Symmetric	Equal to its own transpose (A = Aᵀ), such as a covariance matrix
Upper / lower triangular	Zeros below / above the diagonal
Orthogonal	Columns are perpendicular unit vectors (QᵀQ = I)

The identity matrix behaves like the number 1: A·I = I·A = A.

Basic operations

Addition and subtraction

Add or subtract entry by entry. The matrices must have the same shape.

⎡ 2 1 ⎤   ⎡ 1 4 ⎤   ⎡ 3 5 ⎤
⎣ 0 3 ⎦ + ⎣ 2 5 ⎦ = ⎣ 2 8 ⎦

Scalar multiplication

Multiply every entry by the scalar: 2 × [[2, 1], [0, 3]] = [[4, 2], [0, 6]].

Transpose

The transpose Aᵀ swaps rows and columns: entry (i, j) moves to (j, i). An m × n matrix becomes n × m.

⎡ 1 2 3 ⎤ᵀ    ⎡ 1 4 ⎤
⎣ 4 5 6 ⎦   = ⎢ 2 5 ⎥
              ⎣ 3 6 ⎦

Properties: (Aᵀ)ᵀ = A, (A + B)ᵀ = Aᵀ + Bᵀ, (AB)ᵀ = BᵀAᵀ.

Matrix multiplication

To multiply A (m × n) by B (n × p):

The inner dimensions must match (the number of columns of A equals the number of rows of B).
The result C is m × p.
Entry cᵢⱼ is the dot product of row i of A with column j of B.

(m × n) · (n × p)  →  (m × p)
    └── must match ──┘

Example:

⎡ 1 2 ⎤ ⎡ 0 1 ⎤   ⎡ 1·0+2·1   1·1+2·0 ⎤   ⎡ 2 1 ⎤
⎣ 3 4 ⎦ ⎣ 1 0 ⎦ = ⎣ 3·0+4·1   3·1+4·0 ⎦ = ⎣ 4 3 ⎦

Properties of matrix multiplication

Not commutative: in general AB ≠ BA (often BA does not even exist).
Associative: (AB)C = A(BC).
Distributive: A(B + C) = AB + AC.
Identity: AI = IA = A.

Matrix-vector multiplication

Multiplying a matrix by a vector produces a new vector, a linear combination of the matrix's columns. This is exactly what a neural-network layer and a linear model do.

⎡ 1 2 ⎤         ⎡  1·1 + 2·(−1) ⎤   ⎡ −1 ⎤
⎢ 3 4 ⎥ ⎡  1 ⎤ = ⎢  3·1 + 4·(−1) ⎥ = ⎢ −1 ⎥
⎣ 5 6 ⎦ ⎣ −1 ⎦   ⎣  5·1 + 6·(−1) ⎦   ⎣ −1 ⎦

Element-wise (Hadamard) product

The Hadamard product multiplies matching entries and is written A ⊙ B. It is not matrix multiplication. In NumPy, A * B is element-wise and A @ B is matrix multiplication.

Determinant

The determinant is a single number computed from a square matrix. For a 2 × 2 matrix:

det ⎡ a b ⎤ = ad − bc
    ⎣ c d ⎦

For a 3 × 3 matrix you expand along a row using 2 × 2 determinants (cofactor expansion).

Geometrically, |det A| is the factor by which the matrix scales areas (or volumes). If det A = 0, the matrix squashes space into a lower dimension and is called singular: it has no inverse.

Inverse

The inverse A⁻¹ of a square matrix satisfies A·A⁻¹ = A⁻¹·A = I. It exists only if det A ≠ 0.

For a 2 × 2 matrix:

⎡ a b ⎤⁻¹      1     ⎡  d −b ⎤
⎣ c d ⎦   = ─────── ⎣ −c  a ⎦
             ad − bc

The inverse "undoes" the transformation: if y = Ax then x = A⁻¹y. Properties: (AB)⁻¹ = B⁻¹A⁻¹.

Rank and trace

Rank: the number of linearly independent rows (or columns). A full-rank n × n matrix is invertible. Low rank means redundant information.
Trace: the sum of the diagonal entries, tr(A) = Σ aᵢᵢ.

Eigenvalues and eigenvectors (a first look)

For a square matrix A, an eigenvector v is a non-zero vector whose direction is not changed by A; it is only stretched by a factor λ (the eigenvalue):

A v = λ v

Example. A = [[2, 1], [1, 2]]. Then A·[1, 1] = [3, 3] = 3·[1, 1], so λ = 3 with eigenvector [1, 1]. Also A·[1, −1] = [1, −1], so λ = 1 with eigenvector [1, −1].

Eigenvalues and eigenvectors are the backbone of PCA (principal component analysis): the eigenvectors of the covariance matrix are the directions of greatest variance, and the eigenvalues tell how much variance each direction captures.

Real-life applications

Computer graphics: rotating, scaling and translating objects with transformation matrices.
Image processing: images are matrices; filters are matrices.
Economics: input-output models of industries.
Networks and graphs: adjacency matrices describe who is connected to whom.
Engineering: solving systems of equations for circuits and structures.

Where it appears in machine learning

Data matrix X (samples × features) is the basic object.
Linear regression: predictions for all samples at once, ŷ = X·w.
Neural networks: each layer computes h = f(W·x + b), and training is mostly huge matrix multiplications on GPUs.
PCA and SVD: dimensionality reduction and compression.
Covariance and correlation matrices: describing relationships between features.
Recommender systems: matrix factorisation breaks a user-item matrix into smaller matrices.

Python implementation

import numpy as np

A = np.array([[2, 1],
              [0, 3]])
B = np.array([[1, 4],
              [2, 5]])

print(A + B)          # [[3 5] [2 8]]
print(2 * A)          # [[4 2] [0 6]]
print(A @ B)          # matrix multiplication: [[ 4 13] [ 6 15]]
print(B @ A)          # different: [[ 2 13] [ 4 17]]
print(A * B)          # element-wise product (NOT matrix multiplication)
print(A.T)            # transpose
print(A.shape)        # (2, 2)

C = np.array([[4, 7],
              [2, 6]])
print(np.linalg.det(C))      # 10.0
print(np.linalg.inv(C))      # [[ 0.6 -0.7] [-0.2  0.4]]
print(np.linalg.matrix_rank(C))
print(np.trace(C))           # 10

# Eigenvalues and eigenvectors
S = np.array([[2, 1],
              [1, 2]])
values, vectors = np.linalg.eig(S)
print(values)                # [3. 1.]
print(vectors)

# Predictions of a linear model: y_hat = X @ w
X = np.array([[1, 2],
              [1, 3],
              [1, 4]])      # first column of 1s is for the intercept
w = np.array([1, 2])
print(X @ w)                 # [5 7 9]

Solved examples

Example 1: Addition, subtraction and scalar multiplication

Let A = [[2, 1], [0, 3]] and B = [[1, 4], [2, 5]]. Find A + B, A − B and 2A.

Solution.

A + B = [[3, 5], [2, 8]]
A − B = [[1, −3], [−2, −2]]
2A = [[4, 2], [0, 6]]

Example 2: Matrix multiplication is not commutative

For the same A and B, find AB and BA.

Solution.

AB = [[2·1 + 1·2, 2·4 + 1·5], [0·1 + 3·2, 0·4 + 3·5]] = [[4, 13], [6, 15]]

BA = [[1·2 + 4·0, 1·1 + 4·3], [2·2 + 5·0, 2·1 + 5·3]] = [[2, 13], [4, 17]]

AB ≠ BA.

Example 3: Shapes

A is 3 × 2 and B is 2 × 4. Can you compute AB and BA? What are the shapes?

Solution. AB is defined because the inner dimensions match (2 and 2), and its shape is 3 × 4. BA is not defined, because B has 4 columns but A has 3 rows.

Example 4: Transpose

Find the transpose of [[1, 2, 3], [4, 5, 6]].

Solution. The 2 × 3 matrix becomes 3 × 2: [[1, 4], [2, 5], [3, 6]].

Example 5: Determinant and inverse

Find the determinant and inverse of C = [[4, 7], [2, 6]].

Solution. det C = 4·6 − 7·2 = 24 − 14 = 10.

C⁻¹ = (1/10) × [[6, −7], [−2, 4]] = [[0.6, −0.7], [−0.2, 0.4]].

Check: the first row of C times the first column of C⁻¹ = 4(0.6) + 7(−0.2) = 2.4 − 1.4 = 1. ✓

Example 6: Matrix-vector product

Compute [[1, 2], [3, 4], [5, 6]] · [1, −1].

Solution. Each row is dotted with the vector: 1 − 2 = −1, 3 − 4 = −1, 5 − 6 = −1. The result is [−1, −1, −1].

Example 7: Eigenvalues

Verify that [1, 1] is an eigenvector of A = [[2, 1], [1, 2]] and find its eigenvalue.

Solution. A·[1, 1] = [2 + 1, 1 + 2] = [3, 3] = 3·[1, 1]. So it is an eigenvector with λ = 3.

Example 8: Linear regression predictions

A linear model uses weights w = [1, 2] (intercept 1, slope 2). The data matrix is X = [[1, 2], [1, 3], [1, 4]]. Compute the predictions.

Solution. X·w = [1 + 4, 1 + 6, 1 + 8] = [5, 7, 9].

Common mistakes

Multiplying matrices whose inner dimensions do not match.
Assuming AB = BA.
Using * instead of @ in NumPy for matrix multiplication.
Trying to invert a singular matrix (determinant 0).
Forgetting that (AB)ᵀ = BᵀAᵀ (the order reverses).

Practice problems

Let A = [[1, 2], [3, 4]] and B = [[2, 0], [1, 2]]. Compute AB.
Find the determinant of [[3, 1], [2, 5]].
Find the inverse of [[2, 1], [1, 1]].
If A is 5 × 3 and B is 3 × 2, what is the shape of AB?
Find the transpose of [[1, 2], [3, 4], [5, 6]].
Is [[1, 2], [2, 4]] invertible? Why or why not?

Answers

[[4, 4], [10, 8]]
3·5 − 1·2 = 13
det = 2·1 − 1·1 = 1, so the inverse is [[1, −1], [−1, 2]]
5 × 2
[[1, 3, 5], [2, 4, 6]]
det = 1·4 − 2·2 = 0, so it is singular and not invertible (the second row is twice the first)

Key takeaways

A matrix is a grid of numbers; in ML, rows are usually samples and columns are features.
Multiplication needs matching inner dimensions, is not commutative, and is built from dot products.
The determinant tells you whether a matrix is invertible; the inverse undoes a transformation.
Eigenvectors and eigenvalues describe the directions a matrix only stretches, and they power PCA.
Neural-network layers and linear models are matrix-vector products.

Matrices