Introduction
The dot product (also called the scalar product or inner product) is one of the most important operations in machine learning. It takes two vectors of the same length and returns a single number that tells you how much the two vectors point in the same direction.
Nearly everything in ML is built from dot products: a neuron computes one, matrix multiplication is made of them, attention in Transformers uses them, and similarity search compares embeddings with them.
Definition
For two vectors a = [a₁, a₂, ..., aₙ] and b = [b₁, b₂, ..., bₙ]:
a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ = Σ aᵢbᵢ
Multiply matching entries, then add everything up.
Example. a = [1, 2, 3] and b = [4, −5, 6]:
a · b = 1×4 + 2×(−5) + 3×6 = 4 − 10 + 18 = 12
The two vectors must have the same dimension, and the result is a scalar (not a vector).
Properties
| Property | Statement |
|---|---|
| Commutative | a · b = b · a |
| Distributive | a · (b + c) = a · b + a · c |
| Scalar multiplication | (k a) · b = k (a · b) |
| Self dot product | a · a = ‖a‖² (the squared length) |
| Non-negativity | a · a ≥ 0, and a · a = 0 only for the zero vector |
Geometric meaning
The dot product also equals:
a · b = ‖a‖ · ‖b‖ · cos(θ)
where θ is the angle between the two vectors. This gives the sign a clear meaning:
| Dot product | Angle θ | Meaning |
|---|---|---|
| Positive | less than 90° | Vectors point in roughly the same direction |
| Zero | exactly 90° | Vectors are orthogonal (perpendicular), completely unrelated in direction |
| Negative | more than 90° | Vectors point in roughly opposite directions |
Rearranging gives a way to compute the angle:
cos(θ) = (a · b) / (‖a‖ ‖b‖)
Orthogonality
Two vectors are orthogonal when a · b = 0. For example, [2, −3] and [3, 2]: 2×3 + (−3)×2 = 6 − 6 = 0. Orthogonal vectors carry completely independent directions. Orthogonal weights and features make optimisation easier, and PCA produces orthogonal components.
Cosine similarity
The cosine similarity is the cosine of the angle between two vectors. It compares direction only and ignores length:
cosine similarity = (a · b) / (‖a‖ ‖b‖)
| Value | Meaning |
|---|---|
| +1 | Same direction (identical orientation) |
| 0 | Orthogonal (unrelated) |
| −1 | Opposite directions |
This is the standard way to compare documents, sentences and embeddings: two articles about the same topic have a similar direction even if one is much longer than the other. Cosine distance = 1 − cosine similarity.
If both vectors are already normalised to unit length, cosine similarity is simply their dot product.
Projection
The dot product measures how much of one vector lies in the direction of another.
- Scalar projection of a onto b: (a · b) / ‖b‖
- Vector projection of a onto b: ((a · b) / (b · b)) × b
Example. Projecting a = [3, 4] onto b = [2, 1]: a · b = 10, b · b = 5, so the projection is (10/5) × [2, 1] = [4, 2].
Projections are used in least-squares regression (the fitted values are a projection of y onto the column space of X) and in PCA (data are projected onto the principal components).
Dot product versus other "products"
| Operation | Result | Notes |
|---|---|---|
| Dot product a · b | Scalar | Sum of element-wise products |
| Element-wise (Hadamard) product a ⊙ b | Vector | Multiply matching entries only, no sum |
| Matrix multiplication AB | Matrix | Each entry is a dot product of a row and a column |
| Cross product a × b | Vector (3D only) | Perpendicular to both; used in physics and graphics |
Real-life applications
- Physics: work = force · displacement.
- Computer graphics: lighting, using the angle between a surface normal and the light direction.
- Information retrieval: ranking documents by cosine similarity with a query.
- Recommendation: score = user vector · item vector.
- Signal processing: correlation of signals.
Where it appears in machine learning
- A neuron: z = w · x + b, a weighted sum of inputs, followed by an activation function.
- Linear and logistic regression: the model output is w · x + b.
- Attention in Transformers: the score between a query and a key is their dot product (scaled).
- Embeddings and semantic search: find the most similar vectors using cosine similarity or dot product.
- Support vector machines and kernels: kernels generalise the dot product.
- Matrix multiplication in every neural-network layer is many dot products at once.
Python implementation
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, -5, 6])
print(np.dot(a, b)) # 12
print(a @ b) # 12 (same thing)
print(np.sum(a * b)) # 12 (element-wise product, then sum)
# Angle between vectors
def cos_sim(u, v):
return (u @ v) / (np.linalg.norm(u) * np.linalg.norm(v))
u = np.array([1, 0])
v = np.array([1, 1])
c = cos_sim(u, v)
print(c) # 0.7071...
print(np.degrees(np.arccos(c))) # 45.0
# Projection of a onto b
a2 = np.array([3, 4])
b2 = np.array([2, 1])
proj = (a2 @ b2) / (b2 @ b2) * b2
print(proj) # [4. 2.]
# A single neuron
w = np.array([0.5, -1.0, 2.0])
x = np.array([4.0, 2.0, 1.0])
bias = 0.5
z = w @ x + bias
print(z) # 2.5
print(1 / (1 + np.exp(-z))) # sigmoid(2.5) = 0.924...
Comparing documents with cosine similarity
from sklearn.metrics.pairwise import cosine_similarity
doc1 = np.array([[3, 0, 1]]) # word counts for three words
doc2 = np.array([[1, 2, 0]])
print(cosine_similarity(doc1, doc2)) # [[0.4243]]
Solved examples
Example 1: Computing a dot product
Find a · b for a = [2, 3] and b = [4, −1].
Solution. 2×4 + 3×(−1) = 8 − 3 = 5. The result is positive, so the vectors point in broadly the same direction.
Example 2: Angle between vectors
Find the angle between a = [1, 0] and b = [1, 1].
Solution. a · b = 1. ‖a‖ = 1 and ‖b‖ = √2.
cos θ = 1 / √2 ≈ 0.7071, so θ = 45°.
Example 3: Testing orthogonality
Are [2, −3] and [3, 2] orthogonal?
Solution. 2×3 + (−3)×2 = 6 − 6 = 0, so yes, they are orthogonal.
Example 4: Projection
Project a = [3, 4] onto b = [1, 0].
Solution. a · b = 3 and b · b = 1, so the projection is 3 × [1, 0] = [3, 0]. The scalar projection is 3: the "shadow" of a on the x-axis.
Example 5: Cosine similarity of documents
Two documents have word-count vectors d₁ = [3, 0, 1] and d₂ = [1, 2, 0]. How similar are they?
Solution. d₁ · d₂ = 3 + 0 + 0 = 3. ‖d₁‖ = √10 ≈ 3.162 and ‖d₂‖ = √5 ≈ 2.236.
Cosine similarity = 3 / (3.162 × 2.236) = 3 / 7.071 ≈ 0.424. They share only moderate similarity.
Example 6: A neuron
A neuron has weights w = [0.5, −1, 2], bias b = 0.5 and inputs x = [4, 2, 1]. Find its output with (a) no activation and (b) a sigmoid activation.
Solution. z = 0.5×4 + (−1)×2 + 2×1 + 0.5 = 2 − 2 + 2 + 0.5 = 2.5.
Sigmoid: σ(2.5) = 1 / (1 + e^(−2.5)) ≈ 0.924.
Example 7: Finding an unknown
Find k so that [1, k, 2] is orthogonal to [3, 4, −1].
Solution. 1×3 + k×4 + 2×(−1) = 3 + 4k − 2 = 0, so 4k = −1 and k = −1/4.
Common mistakes
- Trying to take the dot product of vectors with different lengths.
- Confusing the dot product (a scalar) with the element-wise product (a vector).
- Forgetting to divide by the norms when you want cosine similarity.
- Assuming a large dot product always means "very similar". It also grows with vector length, so normalise first when comparing direction.
Practice problems
- Compute [1, 2, 3] · [4, −5, 6].
- Find the angle between [1, 1] and [−1, 1].
- What is the cosine similarity of [1, 2] and [2, 4]?
- Project [5, 0] onto [1, 1].
- Find k such that [k, 3] is orthogonal to [4, −2].
- A neuron has w = [2, −1], x = [3, 4] and b = 1. Compute z = w · x + b.
Answers
- 4 − 10 + 18 = 12
- Dot product = −1 + 1 = 0, so the angle is 90°
- The second vector is twice the first (same direction), so the similarity is 1
- a · b = 5 and b · b = 2, so (5/2) × [1, 1] = [2.5, 2.5]
- 4k − 6 = 0, so k = 1.5
- 6 − 4 + 1 = 3
Key takeaways
- The dot product multiplies matching entries and sums them: a · b = Σ aᵢbᵢ = ‖a‖‖b‖cos θ.
- Positive, zero and negative results mean the vectors are aligned, orthogonal or opposed.
- Cosine similarity normalises away length and compares only direction.
- Projections use the dot product to measure how much of one vector lies along another.
- Neurons, linear models, attention and similarity search are all built from dot products.