Part 3 · 1 chapters · ~8 min

Eigenvectors

Eigenvectors and eigenvalues, power iteration computed, symmetric matrices and orthogonal eigenvectors, PageRank as an eigenvector, principal component analysis for reducing embedding dimensions, the singular value decomposition, and low-rank approximation.

Power iteration in code

code

const A = [[2, 1], [1, 2]]; let x = [1, 0];
for (let i = 0; i < 50; i++) { const y = matvec(A, x); const n = norm(y); x = y.map(t => t / n); }
x                            // [0.7071, 0.7071]
dot(x, matvec(A, x))         // 3: the dominant eigenvalue

# PCA with NumPy: project 1,536-d embeddings to 2-d for a plot
X = embeddings - embeddings.mean(axis=0)
U, S, Vt = np.linalg.svd(X, full_matrices=False)       # SVD: rows of Vt are principal directions
xy = X @ Vt[:2].T                                       # coordinates along the top two components
explained = (S[:2] ** 2).sum() / (S ** 2).sum()         # share of variance kept

SVD factors any matrix into rotation, scaling and rotation (U Σ Vᵀ). Keeping only the largest singular values gives the best low-rank approximation: the idea behind compression of embeddings, recommendation systems (matrix factorisation) and LoRA.

EIGENVECTORS BY POWER ITERATION

directions a matrix only stretches

swipe the figure sideways, or tap expand for full screen

1/4

definition

An eigenvector v of A satisfies A v = λ v: the matrix only scales it, by the eigenvalue λ, without turning it.

A v = λ vstretched, not rotated