Skip to content

4. Linear algebra

Intermediate · 9 min read

You don't need a maths degree for GenAI — but three ideas come up constantly: the dot product, matrix multiplication and the length (norm) of a vector.

4.1 Dot product — how similar are two vectors?

Multiply matching values and add them up. Bigger means "pointing the same way".

import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(a @ b)                 # 1*4 + 2*5 + 3*6
print(np.dot(a, b))          # same thing
print((a * b).sum())         # same thing, spelled out
Output
32
32
32

4.2 Matrix multiplication with @

(rows × k) @ (k × cols) → (rows × cols). The inner sizes must match.

A = np.array([[1, 2],
              [3, 4]])          # 2 x 2
B = np.array([[5, 6, 7],
              [8, 9, 10]])      # 2 x 3
print((A @ B).shape)
print(A @ B)
Output
(2, 3)
[[21 24 27]
 [47 54 61]]

In GenAI: many documents × one query in a single step.

docs = np.array([[1.0, 0.0],
                 [0.7, 0.7],
                 [0.0, 1.0]])         # 3 documents, 2 dimensions
query = np.array([1.0, 0.2])
print(docs @ query)                   # one score per document
Output
[1.   0.84 0.2 ]

* is not matrix multiplication

A * B multiplies element by element; use A @ B for matrix multiplication.

4.3 Vector length (norm) and normalising

v = np.array([3.0, 4.0])
print(np.linalg.norm(v))              # √(3² + 4²)
unit = v / np.linalg.norm(v)          # same direction, length 1
print(unit, np.linalg.norm(unit))

M = np.array([[3.0, 4.0], [6.0, 8.0]])
lengths = np.linalg.norm(M, axis=1, keepdims=True)   # one length per row
print(M / lengths)                    # every row normalised
Output
5.0
[0.6 0.8] 1.0
[[0.6 0.8]
 [0.6 0.8]]

Normalise once, then use plain dot products

For unit-length vectors, the dot product is the cosine similarity. Normalise your stored embeddings once, and every search becomes a single fast @.

4.4 Combining arrays

x = np.array([1, 2])
y = np.array([3, 4])
print(np.concatenate([x, y]))         # join end to end
print(np.vstack([x, y]))              # stack as rows
print(np.hstack([x, y]))              # side by side
print(np.stack([x, y], axis=1))       # pair them as columns
Output
[1 2 3 4]
[[1 2]
 [3 4]]
[1 2 3 4]
[[1 3]
 [2 4]]

4.5 A few more you'll see

M = np.array([[2.0, 1.0], [1.0, 3.0]])
print(np.linalg.inv(M).round(3))              # inverse
print(np.linalg.solve(M, np.array([5.0, 10.0])))   # solve M @ x = b
print(np.linalg.det(M).round(3))              # determinant
Output
[[ 0.6 -0.2]
 [-0.2  0.4]]
[1. 3.]
5.0

Practice

  • Multiply a (4, 3) matrix by a (3,) vector — what shape is the result?
  • Normalise every row of np.random.default_rng(0).normal(size=(5, 8)) and check each row's norm is 1.

Next: NumPy for GenAI →