MNIST Digits Reconstruction by PCA
Interactive demonstration of Principal Component Analysis (PCA) for data compression, following Example 10.4 of Mathematics for Machine Learning (Deisenroth, Faisal & Ong). Each MNIST digit image is a 28×28 = 784-dimensional vector. For each digit class, we collect all its training images into a data matrix and compute the PCA of that matrix. The first k principal components capture the dominant variation; reconstructing an image from only those components shows how a small number of dimensions can represent a high-dimensional digit.
Digit 0 — Original vs Reconstructed
Each row shows one example image of the selected digit. The left column is the original 784-dimensional image; the right column is its reconstruction from the first k principal components of that digit's data matrix.
Explained Variance by Principal Component
The fraction of total variance captured by each principal component (and the cumulative fraction) for the selected digit. The first few components explain most of the variation, which is why a small k reconstructs the digit well.
Mathematical Background
Each MNIST image is a 28×28 grid of pixel intensities, flattened into a vector \(x \in \mathbb{R}^{784}\). For a fixed digit class, we stack all \(n\) training images into a data matrix \(X \in \mathbb{R}^{n \times 784}\). PCA finds an orthonormal basis of principal components that captures the directions of greatest variance in the data.
The data is centered by subtracting the mean image \(\mu\). The principal components are the eigenvectors of the covariance matrix \(C = \frac{1}{n} \bar{X}^T \bar{X}\), where \(\bar{X}\) is the centered data matrix. Equivalently, they are the right singular vectors of \(\bar{X}\).
Any image \(x\) can be projected onto the first \(k\) principal components to get coefficients \(c_i = (x - \mu) \cdot \mathbf{v}_i\), then reconstructed as \(\hat{x} = \mu + \sum_{i=1}^{k} c_i \mathbf{v}_i\). The reconstruction error \(\lVert x - \hat{x} \rVert\) decreases as \(k\) grows. With \(k\) components, each image is represented by just \(k\) numbers instead of 784 — a large compression, at the cost of some detail.
This is exactly the setup of Example 10.4 in the textbook: PCA applied to the MNIST digit data for dimensionality reduction and reconstruction.
Reference: Deisenroth, M.P., Faisal, A.A., Ong, C.S. Mathematics for Machine Learning, Chapter 10 "Dimensionality Reduction with Principal Component Analysis", Example 10.4. Cambridge University Press, 2020. Freely available at mml-book.github.io.
Data: MNIST database of handwritten digits (LeCun, Cortes & Burges), downloaded from the ossci-datasets S3 mirror.