Introduction
Linear algebra is the mathematics of vectors and matrices, and it is the working language of machine learning. Every dataset, every model parameter and every prediction is stored and processed as vectors and matrices. Understanding vectors first makes everything that follows (matrices, dot products, PCA, neural networks) far easier.
What is a vector?
A vector is an ordered list of numbers. It can be viewed in three ways:
- As a list of numbers: v = [3, 4].
- As a point in space: the point (3, 4) on a plane.
- As an arrow from the origin to that point, with a length and a direction.
The number of entries is the dimension. [3, 4] is a 2-dimensional vector; [1.7, 65, 29] is 3-dimensional. A vector with 1,000 entries is simply a point in 1,000-dimensional space, and the same rules apply.
Notation
- Vectors are written in bold or with an arrow: v or v⃗.
- A column vector stands vertically; a row vector lies horizontally. In ML, vectors are usually treated as columns.
- The i-th entry is written vᵢ.
⎡ 3 ⎤
v = ⎢ 4 ⎥ (a 3-dimensional column vector; vᵀ = [3, 4, 5] is its transpose, a row vector)
⎣ 5 ⎦
Special vectors
| Vector | Description | Example (2D) |
|---|---|---|
| Zero vector | All entries are 0 | [0, 0] |
| Unit vector | Has length 1 | [0.6, 0.8] |
| Standard basis vectors | A 1 in one position, 0 elsewhere | e₁ = [1, 0], e₂ = [0, 1] |
| One-hot vector | Encodes a category | "green" in [red, green, blue] → [0, 1, 0] |
Vector operations
Addition and subtraction
Add or subtract entry by entry (the vectors must have the same dimension).
[2, −1, 3] + [1, 4, −2] = [3, 3, 1]
[2, −1, 3] − [1, 4, −2] = [1, −5, 5]
Geometrically, addition places the second arrow at the tip of the first (the "head-to-tail" rule).
Scalar multiplication
Multiply every entry by a number (a scalar). This stretches or shrinks the vector; a negative scalar also flips its direction.
3 × [2, −1, 3] = [6, −3, 9]
Properties
- Commutative: a + b = b + a
- Associative: (a + b) + c = a + (b + c)
- Distributive: k(a + b) = ka + kb
Linear combinations
A linear combination of vectors is a sum of scalar multiples:
c₁·v₁ + c₂·v₂ + ... + cₖ·vₖ
A neural-network neuron computes exactly this kind of combination, and so does linear regression.
The length of a vector: norms
A norm measures the size of a vector.
| Norm | Formula | Also called |
|---|---|---|
| L2 norm | ‖v‖₂ = √(v₁² + v₂² + ... + vₙ²) | Euclidean norm, ordinary length |
| L1 norm | ‖v‖₁ = |v₁| + |v₂| + ... + |vₙ| | Manhattan norm |
| L∞ norm | ‖v‖∞ = max |vᵢ| | Max norm |
| Lp norm | (Σ |vᵢ|ᵖ)^(1/p) | General form |
Example. For v = [1, −2, 2]:
- L2: √(1 + 4 + 4) = 3
- L1: 1 + 2 + 2 = 5
- L∞: 2
Unit vectors and normalisation
Dividing a vector by its L2 norm gives a unit vector in the same direction:
û = v / ‖v‖ e.g. [6, 8] / 10 = [0.6, 0.8]
Distance between vectors
The distance between two points a and b is the norm of their difference.
| Distance | Formula | Intuition |
|---|---|---|
| Euclidean | ‖a − b‖₂ | Straight-line distance |
| Manhattan | ‖a − b‖₁ | Distance walking along a grid of streets |
| Chebyshev | ‖a − b‖∞ | Largest single-coordinate difference |
Linear independence, span and basis
- A set of vectors is linearly independent if none of them can be built as a combination of the others. [1, 0] and [0, 1] are independent; [1, 2] and [2, 4] are dependent because [2, 4] = 2 × [1, 2].
- The span of a set of vectors is everything you can reach with their linear combinations.
- A basis is a set of linearly independent vectors that spans the whole space. The number of basis vectors equals the dimension.
Linear dependence in features (for example two columns that are copies or multiples of each other) is called redundancy or multicollinearity and causes trouble for many models.
Real-life applications
- Physics and engineering: velocity, force and displacement are vectors.
- Computer graphics and games: positions, directions and colours (RGB) are vectors.
- Navigation and GPS: position and movement in space.
- Search engines and recommendations: documents and users represented as vectors.
- Finance: a portfolio's holdings as a vector of weights.
Where it appears in machine learning
- Feature vector: one data sample, for example [age, income, height] = [29, 52000, 1.7].
- Weight vector w: the learned parameters of a linear model.
- Embeddings: words, images and users mapped to vectors (often 128 to 4096 dimensions), so that similar items lie close together.
- Regularisation: L1 (Lasso) and L2 (Ridge) penalties are the L1 and L2 norms of the weight vector.
- k-nearest neighbours and k-means rely on distances between vectors.
- Normalisation: scaling embeddings to unit length before comparing them.
Python implementation
import numpy as np
a = np.array([2, -1, 3])
b = np.array([1, 4, -2])
print(a + b) # [3 3 1]
print(a - b) # [ 1 -5 5]
print(3 * a) # [ 6 -3 9]
print(2 * a - b) # [ 3 -6 8]
v = np.array([1, -2, 2])
print(np.linalg.norm(v)) # 3.0 (L2)
print(np.linalg.norm(v, ord=1)) # 5.0 (L1)
print(np.linalg.norm(v, ord=np.inf)) # 2.0 (max norm)
u = np.array([6, 8])
print(u / np.linalg.norm(u)) # [0.6 0.8] unit vector
# Distances between two points
p = np.array([1, 2, 3])
q = np.array([4, 6, 3])
print(np.linalg.norm(p - q)) # 5.0 Euclidean
print(np.sum(np.abs(p - q))) # 7 Manhattan
print(np.max(np.abs(p - q))) # 4 Chebyshev
Nearest neighbour with vectors
query = np.array([2, 3])
points = {"A": np.array([3, 3]), "B": np.array([6, 7]), "C": np.array([2, 5])}
distances = {name: np.linalg.norm(query - p) for name, p in points.items()}
print(distances) # {'A': 1.0, 'B': 5.0, 'C': 2.0}
print(min(distances, key=distances.get)) # A
Solved examples
Example 1: Basic operations
Let a = [2, −1, 3] and b = [1, 4, −2]. Compute a + b, a − b, 3a and 2a − b.
Solution.
- a + b = [3, 3, 1]
- a − b = [1, −5, 5]
- 3a = [6, −3, 9]
- 2a − b = [4 − 1, −2 − 4, 6 + 2] = [3, −6, 8]
Example 2: Norms
Find the L1, L2 and L∞ norms of v = [1, −2, 2].
Solution. L1 = 1 + 2 + 2 = 5. L2 = √(1 + 4 + 4) = √9 = 3. L∞ = max(1, 2, 2) = 2.
Example 3: Unit vector
Normalise v = [6, 8].
Solution. ‖v‖ = √(36 + 64) = 10. Unit vector = [6/10, 8/10] = [0.6, 0.8]. Check: √(0.36 + 0.64) = 1.
Example 4: Distances
Find the Euclidean, Manhattan and Chebyshev distances between (1, 2, 3) and (4, 6, 3).
Solution. The difference is (3, 4, 0).
- Euclidean = √(9 + 16 + 0) = 5
- Manhattan = 3 + 4 + 0 = 7
- Chebyshev = 4
Example 5: Linear independence
Are [1, 2] and [2, 4] linearly independent? Are [1, 0] and [0, 1]?
Solution. [2, 4] = 2 × [1, 2], so the first pair is dependent (they lie on the same line). No scalar multiple of [1, 0] gives [0, 1], so the second pair is independent.
Example 6: k-nearest neighbour
A new point is q = [2, 3]. Among A = [3, 3], B = [6, 7] and C = [2, 5], which is nearest?
Solution. Distances: to A = √(1 + 0) = 1; to B = √(16 + 16) ≈ 5.66; to C = √(0 + 4) = 2. The nearest point is A.
Common mistakes
- Adding vectors of different dimensions.
- Confusing the norm of a vector (a single number) with the vector itself.
- Forgetting to take the square root when computing an L2 norm.
- Forgetting that "normalising" a vector (dividing by its norm) is different from standardising a feature (subtracting the mean and dividing by the standard deviation).
Practice problems
- Let u = [1, 2] and v = [3, −1]. Compute 2u + 3v.
- Find the L2 norm of [5, 12].
- Find the unit vector in the direction of [2, −2, 1].
- Find the Manhattan distance between [1, 5] and [4, 1].
- Are [1, 2, 3] and [2, 4, 6] linearly independent?
- Write the one-hot vector for "green" if the categories are [red, green, blue].
Answers
- [2, 4] + [9, −3] = [11, 1]
- √(25 + 144) = √169 = 13
- ‖v‖ = √(4 + 4 + 1) = 3, so [2/3, −2/3, 1/3]
- |1 − 4| + |5 − 1| = 3 + 4 = 7
- No, the second is 2 times the first (linearly dependent)
- [0, 1, 0]
Key takeaways
- A vector is an ordered list of numbers: a point or an arrow in n-dimensional space.
- You can add vectors and scale them; combinations of these two operations are called linear combinations.
- Norms measure length (L1, L2, L∞) and the norm of a difference measures distance.
- Independence and basis describe how much genuinely new information a set of vectors carries.
- In ML, data points, weights and embeddings are all vectors.