Computer vision
A convolutional network that reads digits you draw, trained with TensorFlow and running in your browser.
99.24%
Test accuracy (browser model)
76
Mistakes out of 10,000
97.0%
Shifted digits
55 KB
Weights
Overview
Reading handwriting was one of the first jobs neural networks did in the real world, sorting post and cheques. It is still the clearest way to show what a convolutional network adds: it learns small visual patterns and combines them, instead of memorising pixels.
This rebuild climbs from a model with no hidden layer to a small CNN trained with augmentation, tests each one on 10,000 digits it never saw and on shifted, rotated and shrunk copies, then shrinks the winner to 8-bit weights so it runs on the canvas below.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
70,000 digits
28×28 grayscale, labelled 0–9
- 02
Prepare
0–1 pixels
55,000 train · 5,000 val · 10,000 test
- 03
Baseline
92.7%
No hidden layer
- 04
CNN
56,714 params
2 conv blocks + dense layer
- 05
Augment
shift · rotate · zoom
Random changes while training
- 06
Ship
99.24%
55 KB of 8-bit weights
Inside the model
Inside a convolutional network.
Scroll to follow one real test digit through every layer: the feature maps below are computed live in your browser with the network's real weights.
28 × 28 = 784 numbers, 0 (paper) to 255 (ink). That is all the network gets.
At every position: 9 pixels × 9 weights, added up. Dark where the picture matches the filter's pattern.
16 filters → 16 feature maps (26 × 26)
max-pool: 26 × 26 → 13 × 13
Each 2 × 2 block keeps only its strongest value: a quarter of the numbers, the same pattern, and a stroke can shift a pixel without changing much.
layer 2: 32 filters, each looking at all 16 maps at once
These combine strokes into parts: corners, loops, crossings. Then another pool: 5 × 5 × 32 = 800 numbers.
800 numbers → 64 neurons → 10 answers
Each dot is one neuron: a weighted sum of everything before it, then “keep it if positive” (ReLU). Darker = more active.
softmax: scores → chances that add up to 100%
The right answer: 2
accuracy after each pass over the training data
■ held-out validation images · ■ training images
Same filter everywhere
A stroke detector learned in one corner works in every corner.
Few weights
A filter has 9 weights per input map, however big the picture.
Layers build up
Edges → parts → whole objects, from one layer to the next.
Nothing hand-made
Every filter here was learned from examples, none designed by me.
■ blue weights subtract, ■ orange add.
The data
70,000 handwritten digits, 28 by 28 pixels.
The classic first dataset of computer vision: small, clean, and still the clearest way to see what a convolutional network learns.
MNIST handwritten digits
Digits written by US Census Bureau employees and high-school students, collected by NIST and normalised by Yann LeCun, Corinna Cortes and Christopher Burges: each one centred in a 28×28 grayscale image. The test set's writers are different people from the training set's. Licence: CC BY-SA 3.0 (Yann LeCun and Corinna Cortes); loaded from Keras' public mirror.
60,000
training images
10,000
test images
28 × 28
pixels each
10
classes
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| pixels | 28×28 · 0–255 | Grayscale image of one digit, white ink on black; scaled to 0–1 |
| label | 0–9 | The digit written: what the network predicts |
Explore the data
Average every digit and the shapes appear.
Thousands of handwritings blur into one ghost per digit. A model with no hidden layer learns little more than these templates.
Close to balanced: from 5,421 fives to 6,742 ones in training.
Share of pixels with ink: 1s use the least (9.4%), 0s the most (21.1%). Simple pixel counts already separate some digits.
Try it
Draw a digit. Watch the network read it.
Your drawing is cropped, scaled and centred the way MNIST digits were, then run through the network on every stroke. Watch the neurons light up and the winning digit's output fire.
Live demo
runs in your browser
Draw 0–9
Inside the network, live
brighter = more active · wires = strongest contributions
The network says
Test image T-02549, labelled 3. Read correctly.
Filters 1: 16 maps, 26×26
Edges, corners and stroke ends.
Filters 2: 32 maps, 11×11
Loops, crossings, the bar of a 7, the tail of a 9.
Or pick a real test image
One random test image per digit, then the six mistakes the network was most sure about (orange).
How well it works
76 mistakes in 10,000 test digits.
99.24% accuracy for the 8-bit model in your browser (99.25% before shrinking), against 92.7% for a model with no hidden layer.
56,714 parameters in total; most of them sit in the dense layer.
Input
28×28×1
your drawing
—
Conv 3×3
26×26×16
16 filters
160 params
Max-pool
13×13×16
keep the strongest
—
Conv 3×3
11×11×32
32 filters
4,640 params
Max-pool
5×5×32
800 numbers
—
Dense
64
dropout while training
51,264 params
Output
10
one score per digit
650 params
On clean test digits the CNNs tie (99.21% vs 99.25%). Move the digits around and augmentation pays: shifted digits 71.3% → 97.0%, shrunk 92.8% → 97.8%.
Stopped after 31 epochs, when validation loss stopped improving. Training accuracy sits below validation because augmentation and dropout make training harder on purpose.
Weakest: 9 (98.41%). Strongest: 4 (99.69%).
Dark diagonal: % read correctly. Orange: mistakes. The most common: 9 read as 4 (7 times).
| true ↓ · read as → | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 99.5 | 2 | 1 | 1 | 1 | |||||
| 1 | 99.0 | 3 | 1 | 3 | 3 | 1 | ||||
| 2 | 1 | 99.2 | 2 | 1 | 3 | 1 | ||||
| 3 | 99.5 | 4 | 1 | |||||||
| 4 | 99.7 | 1 | 1 | 1 | ||||||
| 5 | 3 | 99.4 | 1 | 1 | ||||||
| 6 | 1 | 2 | 2 | 99.4 | 1 | |||||
| 7 | 3 | 2 | 99.3 | 1 | 1 | |||||
| 8 | 2 | 3 | 1 | 1 | 1 | 1 | 99.0 | 1 | ||
| 9 | 1 | 7 | 2 | 2 | 4 | 98.4 |
Some are genuinely ambiguous; a few look mislabelled. This is where 99% stops being about the model and starts being about the data.
label 5 · read 6
100% sure
label 9 · read 4
100% sure
label 3 · read 5
100% sure
label 2 · read 0
99% sure
label 0 · read 4
99% sure
label 9 · read 8
99% sure
Trained on CPU with early stopping; seconds are wall-clock time on a laptop.
| Model | Parameters | Test accuracy | Shifted | Rotated | Shrunk | Epochs | Seconds |
|---|---|---|---|---|---|---|---|
| Softmax regression (no hidden layer) | 7,850 | 92.74% | 33.4% | 83.8% | 37.2% | 40 | 30.7 |
| Dense network (1 hidden layer) | 101,770 | 98.06% | 47.7% | 93.5% | 61.5% | 24 | 32.2 |
| Small CNN | 56,714 | 99.21% | 71.3% | 97.0% | 92.8% | 19 | 188 |
| Small CNN + augmentation (shipped) | 56,714 | 99.25% | 97.0% | 98.3% | 97.8% | 31 | 395.4 |
| Shipped, 8-bit weights | 56,714 | 99.24% | – | – | – | – | – |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
mnist() from ml/train.py + 2 helpers (r, save)
pip install numpy pandas scikit-learn scipy tensorflow · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load MNIST and hold out validation
Seeds TensorFlow for repeatable runs, loads mnist.npz, sets aside 5000 random training images for validation and scales pixels to 0–1.
ml/train.py:2856 · 18 lines
import base64 import time import keras import tensorflow as tf from keras import layers from scipy import ndimage print("handwritten-digits") keras.utils.set_random_seed(SEED) tf.config.experimental.enable_op_determinism() z = np.load(RAW / "mnist" / "mnist.npz", allow_pickle=False) xtr_all, ytr_all, xte, yte = z["x_train"], z["y_train"], z["x_test"], z["y_test"] rng = np.random.default_rng(SEED) perm = rng.permutation(len(xtr_all)) val_idx, tr_idx = perm[:5000], perm[5000:] f = lambda a: (a.astype("float32") / 255.0)[..., None] # noqa: E731 Xtr, ytr, Xva, yva, Xte = f(xtr_all[tr_idx]), ytr_all[tr_idx], f(xtr_all[val_idx]), ytr_all[val_idx], f(xte) - 2
Define four candidate networks
From simple to strong: softmax regression, a dense net with one 128-unit hidden layer, a small two-conv CNN, and the same CNN with random rotation, shift and zoom.
ml/train.py:2876 · 30 lines
def softmax_reg(): return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(10)], name="softmax") def mlp(): return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(128, activation="relu"), layers.Dropout(0.2), layers.Dense(10)], name="mlp") def cnn(augment: bool): aug = [layers.RandomRotation(0.04, fill_mode="constant"), layers.RandomTranslation(0.1, 0.1, fill_mode="constant"), layers.RandomZoom(0.1, fill_mode="constant")] if augment else [] return keras.Sequential( [ keras.Input((28, 28, 1)), *aug, layers.Conv2D(16, 3, activation="relu", name="conv1"), layers.MaxPooling2D(2), layers.Conv2D(32, 3, activation="relu", name="conv2"), layers.MaxPooling2D(2), - 3
Build distorted test sets
Copies of the test digits shifted up to 4 px, rotated up to 25 degrees or shrunk to 70%, to check which model copes with messier handwriting.
ml/train.py:2908 · 6 lines
# robustness sets: the same test digits, shifted, rotated or shrunk (seeded) rr = np.random.default_rng(SEED + 1) shift = np.stack([ndimage.shift(im, (rr.integers(-4, 5), rr.integers(-4, 5)), order=1, mode="constant") for im in xte.astype("float32")]) rot = np.stack([ndimage.rotate(im, rr.uniform(-25, 25), reshape=False, order=1, mode="constant") for im in xte.astype("float32")]) small = np.stack([np.pad(ndimage.zoom(im, 0.7, order=1), ((4, 4), (4, 4)))[:28, :28] for im in xte.astype("float32")]) robust_sets = {"Original": Xte, "Shifted ±4 px": f(np.clip(shift, 0, 255)), "Rotated ±25°": f(np.clip(rot, 0, 255)), "Shrunk to 70%": f(np.clip(small, 0, 255))} - 4
Train and compare the networks
Each network trains up to 40 epochs with Adam, early stopping and learning-rate halving on validation loss, then is scored on the original and distorted test sets.
ml/train.py:2916 · 15 lines
comparison, histories, trained = [], {}, {} for name, build in builders.items(): model = build() model.compile(optimizer=keras.optimizers.Adam(1e-3), loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"]) t0 = time.time() h = model.fit( Xtr, ytr, validation_data=(Xva, yva), epochs=40, batch_size=128, verbose=0, callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True, monitor="val_loss"), keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=2, min_lr=1e-5)], ) secs = time.time() - t0 rob = {k: r(float(np.mean(model.predict(v, verbose=0, batch_size=1000).argmax(1) == yte)), 4) for k, v in robust_sets.items()} comparison.append({"model": name, "params": int(model.count_params()), "acc": rob["Original"], "robust": rob, "epochs": len(h.history["loss"]), "seconds": r(secs, 1)}) histories[name] = {k: [r(v, 6 if k == "learning_rate" else 4) for v in vals] for k, vals in h.history.items()} trained[name] = model print(f" {name:38s} params={model.count_params():6d} acc={rob['Original']:.4f} shift={rob['Shifted ±4 px']:.4f} rot={rob['Rotated ±25°']:.4f} small={rob['Shrunk to 70%']:.4f} epochs={len(h.history['loss'])} {secs:.0f}s") - 5
Quantise to int8 and re-test
Keeps the augmented CNN, stores each layer as int8 with one scale per output unit, and re-scores the test set with the NumPy forward pass the browser mirrors.
ml/train.py:2933 · 49 lines
chosen = "Small CNN + augmentation" model = trained[chosen] # export with int8 weights (one scale per output channel / unit), evaluated with the browser's forward pass def q8(w, axis_out): moved = np.moveaxis(w, axis_out, -1) scale = np.abs(moved).reshape(-1, moved.shape[-1]).max(axis=0) / 127 scale = np.array([float(f"{s:.8g}") for s in scale]) q = np.round(moved / scale).astype(np.int8) return q, scale L = {ly.name: ly.get_weights() for ly in model.layers if ly.get_weights()} export, deq = {}, {} for name in ["conv1", "conv2", "dense1", "out"]: w, b = L[name] q, scale = q8(w, w.ndim - 1) - 6
Analyse errors, pick demo digits
Counts the confusion matrix and top 8 mix-ups, picks a random test digit per class plus the 6 most confident mistakes, and averages each digit's training images.
ml/train.py:2984 · 17 lines
conf = [[int(((yte == a) & (pred == p)).sum()) for p in range(10)] for a in range(10)] pairs = sorted([(conf[a][p], a, p) for a in range(10) for p in range(10) if a != p], reverse=True)[:8] per_digit = [r(conf[d][d] / sum(conf[d]), 4) for d in range(10)] def b64img(im): return base64.b64encode(np.asarray(im, dtype=np.uint8).tobytes()).decode("ascii") rs = np.random.default_rng(SEED + 2) pick = [int(rs.choice(np.where(yte == dgt)[0])) for dgt in range(10)] wrong = np.where(pred != yte)[0] worst = [int(i) for i in wrong[np.argsort(-prob[wrong].max(1))][:6]] samples = [ {"id": f"T-{i:05d}", "label": int(yte[i]), "pred": int(pred[i]), "p": r(prob[i].max(), 4), "probs": [r(v, 5) for v in prob[i]], "img": b64img(xte[i]), "kind": "mistake" if i in worst else "random"} for i in pick + worst ] means = [b64img(np.round(xtr_all[ytr_all == dgt].mean(axis=0))) for dgt in range(10)] ink = [r(float((xtr_all[ytr_all == dgt] > 50).mean()), 4) for dgt in range(10)] - 7
Export the JSON
Writes the int8 weights, model comparison, training histories, confusion matrix, digit averages and samples to src/data/models/handwritten-digits.json.
ml/train.py:3003 · 29 lines
save( "handwritten-digits", { "dataset": { "name": "MNIST handwritten digits", "source": "https://yann.lecun.com/exdb/mnist/", "download": "https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz", "license": "CC BY-SA 3.0 (Yann LeCun and Corinna Cortes)", "train": int(len(xtr_all)), "test": int(len(xte)), "size": 28, }, "split": {"train": int(len(tr_idx)), "val": int(len(val_idx)), "test": int(len(xte))}, "counts": {"train": np.bincount(ytr_all).tolist(), "test": np.bincount(yte).tolist()}, "comparison": comparison, "chosen": chosen,
In your browser · TypeScript
- 8
Unpack the 8-bit weights
Each weight was stored as an int8 with one scale per output channel; multiplying back gives the float the network uses.
src/lib/cnn.ts:9 · 7 lines
export function dequant(l: QLayer): Float64Array { const bin = atob(l.q); const out = l.shape[l.shape.length - 1]; const w = new Float64Array(bin.length); for (let i = 0; i < bin.length; i++) w[i] = ((bin.charCodeAt(i) << 24) >> 24) * l.scale[i % out]; return w; } - 9
The CNN, layer by layer, in TypeScript
Convolution, ReLU, pooling, dense layers and softmax, returning every intermediate activation for the neuron view.
src/lib/cnn.ts:93 · 27 lines
export function createDigitNet(m: DigitCnn) { const W1 = dequant(m.conv1); const W2 = dequant(m.conv2); const W3 = dequant(m.dense1); const W4 = dequant(m.out); const c1 = m.conv1.shape[3]; const c2 = m.conv2.shape[3]; const d1 = m.dense1.shape[1]; const d2 = m.out.shape[1]; /** `pixels`: 784 values in 0–1, row by row. Returns class probabilities and every intermediate activation. */ function run(pixels: ArrayLike<number>) { const input: Tensor = { data: Float64Array.from(pixels), h: 28, w: 28, c: 1 }; const a1 = conv(input, W1, m.conv1.b, c1); const p1 = pool(a1); const a2 = conv(p1, W2, m.conv2.b, c2);
Notes
The decisions behind it.
Approach
- 1
Kept the official 10,000-image test set for the end; held back 5,000 training images for early stopping.
- 2
Trained four networks of growing power with TensorFlow/Keras: no hidden layer, one dense hidden layer, a small CNN, and the same CNN with augmentation.
- 3
Tested every model on the plain test set and on shifted, rotated and shrunk copies of it.
- 4
Quantised the chosen CNN to 8 bits per weight and re-tested it with the exact forward pass the browser runs.
Test accuracy · shifted digits
Honest note
MNIST is a solved benchmark: its digits are clean, centred and written by a few thousand people decades ago, so 99.24% on its test set says more about the method than about real-world handwriting. Digits drawn with a mouse look different, which is exactly why the shipped model was trained with augmentation; it still slips on very thin, off-centre or unusual digits. The plain CNN scores 99.21% on the test set, close to the augmented one, but falls to 71.3% on shifted digits.
Reproduce: ml/train.py → mnist()
dataset sourceNext project
Hand Gestures