←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

A convolutional network that reads digits you draw, trained with TensorFlow and running in your browser.

99.24%

Test accuracy (browser model)

76

Mistakes out of 10,000

97.0%

Shifted digits

55 KB

Weights

Overview

Reading handwriting was one of the first jobs neural networks did in the real world, sorting post and cheques. It is still the clearest way to show what a convolutional network adds: it learns small visual patterns and combines them, instead of memorising pixels.

This rebuild climbs from a model with no hidden layer to a small CNN trained with augmentation, tests each one on 10,000 digits it never saw and on shifted, rotated and shrunk copies, then shrinks the winner to 8-bit weights so it runs on the canvas below.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    70,000 digits

    28×28 grayscale, labelled 0–9

  2. 02

    Prepare

    0–1 pixels

    55,000 train · 5,000 val · 10,000 test

  3. 03

    Baseline

    92.7%

    No hidden layer

  4. 04

    CNN

    56,714 params

    2 conv blocks + dense layer

  5. 05

    Augment

    shift · rotate · zoom

    Random changes while training

  6. 06

    Ship

    99.24%

    55 KB of 8-bit weights

Inside the model

Inside a convolutional network.

Scroll to follow one real test digit through every layer: the feature maps below are computed live in your browser with the network's real weights.

digit
2522025225224646252252252252249157252252252252253252252252252252253252252252252252253252224167167208253252

28 × 28 = 784 numbers, 0 (paper) to 255 (ink). That is all the network gets.

0.330.20-0.050.100.100.200.030.260.29
filter 11 of 16→

At every position: 9 pixels × 9 weights, added up. Dark where the picture matches the filter's pattern.

16 filters → 16 feature maps (26 × 26)

max-pool: 26 × 26 → 13 × 13

→

Each 2 × 2 block keeps only its strongest value: a quarter of the numbers, the same pattern, and a stroke can shift a pixel without changing much.

layer 2: 32 filters, each looking at all 16 maps at once

These combine strokes into parts: corners, loops, crossings. Then another pool: 5 × 5 × 32 = 800 numbers.

800 numbers → 64 neurons → 10 answers

→
217863

Each dot is one neuron: a weighted sum of everything before it, then “keep it if positive” (ReLU). Darker = more active.

softmax: scores → chances that add up to 100%

2
100.0%
1
0.0%
7
0.0%
8
0.0%
6
0.0%

The right answer: 2

accuracy after each pass over the training data

100%78%31 EPOCHS

■ held-out validation images · ■ training images

Same filter everywhere

A stroke detector learned in one corner works in every corner.

Few weights

A filter has 9 weights per input map, however big the picture.

Layers build up

Edges → parts → whole objects, from one layer to the next.

Nothing hand-made

Every filter here was learned from examples, none designed by me.

■ blue weights subtract, ■ orange add.

The data

70,000 handwritten digits, 28 by 28 pixels.

The classic first dataset of computer vision: small, clean, and still the clearest way to see what a convolutional network learns.

MNIST handwritten digits

Digits written by US Census Bureau employees and high-school students, collected by NIST and normalised by Yann LeCun, Corinna Cortes and Christopher Burges: each one centred in a 28×28 grayscale image. The test set's writers are different people from the training set's. Licence: CC BY-SA 3.0 (Yann LeCun and Corinna Cortes); loaded from Keras' public mirror.

60,000

training images

10,000

test images

28 × 28

pixels each

10

classes

MNIST (Yann LeCun's page)

Column dictionary

● used by the model

ColumnTypeWhat it means
pixels28×28 · 0–255Grayscale image of one digit, white ink on black; scaled to 0–1
label0–9The digit written: what the network predicts

Explore the data

Average every digit and the shapes appear.

The average image of each digit

Thousands of handwritings blur into one ghost per digit. A model with no hidden layer learns little more than these templates.

0
1
2
3
4
5
6
7
8
9
Images per digit

Close to balanced: from 5,421 fives to 6,742 ones in training.

05,00010,00006,74212345,42156789
Ink per digit

Share of pixels with ink: 1s use the least (9.4%), 0s the most (21.1%). Simple pixel counts already separate some digits.

0.0%25.0%50.0%21.1%09.4%123456789

Try it

Draw a digit. Watch the network read it.

Your drawing is cropped, scaled and centred the way MNIST digits were, then run through the network on every stroke. Watch the neurons light up and the winning digit's output fire.

Live demo

runs in your browser

Draw 0–9

Inside the network, live

inputfilters 1filters 2denseoutput00.0%10.0%20.0%3100%40.0%50.0%60.0%70.0%80.0%90.0%

The network says

3100.0% sure
3
100.0%
5
0.0%
9
0.0%

Test image T-02549, labelled 3. Read correctly.

Filters 1: 16 maps, 26×26

Edges, corners and stroke ends.

Filters 2: 32 maps, 11×11

Loops, crossings, the bar of a 7, the tail of a 9.

Or pick a real test image

One random test image per digit, then the six mistakes the network was most sure about (orange).

How well it works

76 mistakes in 10,000 test digits.

99.24% accuracy for the 8-bit model in your browser (99.25% before shrinking), against 92.7% for a model with no hidden layer.

The network, layer by layer

56,714 parameters in total; most of them sit in the dense layer.

Input

28×28×1

your drawing

—

Conv 3×3

26×26×16

16 filters

160 params

Max-pool

13×13×16

keep the strongest

—

Conv 3×3

11×11×32

32 filters

4,640 params

Max-pool

5×5×32

800 numbers

—

Dense

64

dropout while training

51,264 params

Output

10

one score per digit

650 params

Four networks, four kinds of test

On clean test digits the CNNs tie (99.21% vs 99.25%). Move the digits around and augmentation pays: shifted digits 71.3% → 97.0%, shrunk 92.8% → 97.8%.

OriginalShifted ±4 pxRotated ±25°Shrunk to 70%
0%50%100%93%33%84%37%No hidden layer98%48%94%62%Dense99%71%97%93%CNN99%97%98%98%CNN + aug.
Training the shipped network

Stopped after 31 epochs, when validation loss stopped improving. Training accuracy sits below validation because augmentation and dropout make training harder on purpose.

Training accuracyValidation accuracyPlain CNN, validation
90%95%100%19162431Epoch
Per-digit accuracy

Weakest: 9 (98.41%). Strongest: 4 (99.69%).

0.0%50.0%100.0%012399.7%4567898.4%9
Which digits get mixed up

Dark diagonal: % read correctly. Orange: mistakes. The most common: 9 read as 4 (7 times).

true ↓ · read as →0123456789
099.52111
199.031331
2199.22131
399.541
499.7111
5399.411
612299.41
73299.311
823111199.01
91722498.4
The mistakes it was surest about

Some are genuinely ambiguous; a few look mislabelled. This is where 99% stops being about the model and starts being about the data.

label 5 · read 6

100% sure

label 9 · read 4

100% sure

label 3 · read 5

100% sure

label 2 · read 0

99% sure

label 0 · read 4

99% sure

label 9 · read 8

99% sure

All networks

Trained on CPU with early stopping; seconds are wall-clock time on a laptop.

ModelParametersTest accuracyShiftedRotatedShrunkEpochsSeconds
Softmax regression (no hidden layer)7,85092.74%33.4%83.8%37.2%4030.7
Dense network (1 hidden layer)101,77098.06%47.7%93.5%61.5%2432.2
Small CNN56,71499.21%71.3%97.0%92.8%19188
Small CNN + augmentation (shipped)56,71499.25%97.0%98.3%97.8%31395.4
Shipped, 8-bit weights56,71499.24%–––––

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

mnist() from ml/train.py + 2 helpers (r, save)

pip install numpy pandas scikit-learn scipy tensorflow · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load MNIST and hold out validation

    Seeds TensorFlow for repeatable runs, loads mnist.npz, sets aside 5000 random training images for validation and scales pixels to 0–1.

    ml/train.py:2856 · 18 lines

    import base64
    import time
    
    import keras
    import tensorflow as tf
    from keras import layers
    from scipy import ndimage
    
    print("handwritten-digits")
    keras.utils.set_random_seed(SEED)
    tf.config.experimental.enable_op_determinism()
    z = np.load(RAW / "mnist" / "mnist.npz", allow_pickle=False)
    xtr_all, ytr_all, xte, yte = z["x_train"], z["y_train"], z["x_test"], z["y_test"]
    rng = np.random.default_rng(SEED)
    perm = rng.permutation(len(xtr_all))
    val_idx, tr_idx = perm[:5000], perm[5000:]
    f = lambda a: (a.astype("float32") / 255.0)[..., None]  # noqa: E731
    Xtr, ytr, Xva, yva, Xte = f(xtr_all[tr_idx]), ytr_all[tr_idx], f(xtr_all[val_idx]), ytr_all[val_idx], f(xte)
  2. 2

    Define four candidate networks

    From simple to strong: softmax regression, a dense net with one 128-unit hidden layer, a small two-conv CNN, and the same CNN with random rotation, shift and zoom.

    ml/train.py:2876 · 30 lines

    def softmax_reg():
        return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(10)], name="softmax")
    
    def mlp():
        return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(128, activation="relu"), layers.Dropout(0.2), layers.Dense(10)], name="mlp")
    
    def cnn(augment: bool):
        aug = [layers.RandomRotation(0.04, fill_mode="constant"), layers.RandomTranslation(0.1, 0.1, fill_mode="constant"), layers.RandomZoom(0.1, fill_mode="constant")] if augment else []
        return keras.Sequential(
            [
                keras.Input((28, 28, 1)),
                *aug,
                layers.Conv2D(16, 3, activation="relu", name="conv1"),
                layers.MaxPooling2D(2),
                layers.Conv2D(32, 3, activation="relu", name="conv2"),
                layers.MaxPooling2D(2),
  3. 3

    Build distorted test sets

    Copies of the test digits shifted up to 4 px, rotated up to 25 degrees or shrunk to 70%, to check which model copes with messier handwriting.

    ml/train.py:2908 · 6 lines

    # robustness sets: the same test digits, shifted, rotated or shrunk (seeded)
    rr = np.random.default_rng(SEED + 1)
    shift = np.stack([ndimage.shift(im, (rr.integers(-4, 5), rr.integers(-4, 5)), order=1, mode="constant") for im in xte.astype("float32")])
    rot = np.stack([ndimage.rotate(im, rr.uniform(-25, 25), reshape=False, order=1, mode="constant") for im in xte.astype("float32")])
    small = np.stack([np.pad(ndimage.zoom(im, 0.7, order=1), ((4, 4), (4, 4)))[:28, :28] for im in xte.astype("float32")])
    robust_sets = {"Original": Xte, "Shifted ±4 px": f(np.clip(shift, 0, 255)), "Rotated ±25°": f(np.clip(rot, 0, 255)), "Shrunk to 70%": f(np.clip(small, 0, 255))}
  4. 4

    Train and compare the networks

    Each network trains up to 40 epochs with Adam, early stopping and learning-rate halving on validation loss, then is scored on the original and distorted test sets.

    ml/train.py:2916 · 15 lines

    comparison, histories, trained = [], {}, {}
    for name, build in builders.items():
        model = build()
        model.compile(optimizer=keras.optimizers.Adam(1e-3), loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"])
        t0 = time.time()
        h = model.fit(
            Xtr, ytr, validation_data=(Xva, yva), epochs=40, batch_size=128, verbose=0,
            callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True, monitor="val_loss"), keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=2, min_lr=1e-5)],
        )
        secs = time.time() - t0
        rob = {k: r(float(np.mean(model.predict(v, verbose=0, batch_size=1000).argmax(1) == yte)), 4) for k, v in robust_sets.items()}
        comparison.append({"model": name, "params": int(model.count_params()), "acc": rob["Original"], "robust": rob, "epochs": len(h.history["loss"]), "seconds": r(secs, 1)})
        histories[name] = {k: [r(v, 6 if k == "learning_rate" else 4) for v in vals] for k, vals in h.history.items()}
        trained[name] = model
        print(f"  {name:38s} params={model.count_params():6d} acc={rob['Original']:.4f} shift={rob['Shifted ±4 px']:.4f} rot={rob['Rotated ±25°']:.4f} small={rob['Shrunk to 70%']:.4f} epochs={len(h.history['loss'])} {secs:.0f}s")
  5. 5

    Quantise to int8 and re-test

    Keeps the augmented CNN, stores each layer as int8 with one scale per output unit, and re-scores the test set with the NumPy forward pass the browser mirrors.

    ml/train.py:2933 · 49 lines

    chosen = "Small CNN + augmentation"
    model = trained[chosen]
    
    # export with int8 weights (one scale per output channel / unit), evaluated with the browser's forward pass
    def q8(w, axis_out):
        moved = np.moveaxis(w, axis_out, -1)
        scale = np.abs(moved).reshape(-1, moved.shape[-1]).max(axis=0) / 127
        scale = np.array([float(f"{s:.8g}") for s in scale])
        q = np.round(moved / scale).astype(np.int8)
        return q, scale
    
    L = {ly.name: ly.get_weights() for ly in model.layers if ly.get_weights()}
    export, deq = {}, {}
    for name in ["conv1", "conv2", "dense1", "out"]:
        w, b = L[name]
        q, scale = q8(w, w.ndim - 1)
  6. 6

    Analyse errors, pick demo digits

    Counts the confusion matrix and top 8 mix-ups, picks a random test digit per class plus the 6 most confident mistakes, and averages each digit's training images.

    ml/train.py:2984 · 17 lines

    conf = [[int(((yte == a) & (pred == p)).sum()) for p in range(10)] for a in range(10)]
    pairs = sorted([(conf[a][p], a, p) for a in range(10) for p in range(10) if a != p], reverse=True)[:8]
    per_digit = [r(conf[d][d] / sum(conf[d]), 4) for d in range(10)]
    
    def b64img(im):
        return base64.b64encode(np.asarray(im, dtype=np.uint8).tobytes()).decode("ascii")
    
    rs = np.random.default_rng(SEED + 2)
    pick = [int(rs.choice(np.where(yte == dgt)[0])) for dgt in range(10)]
    wrong = np.where(pred != yte)[0]
    worst = [int(i) for i in wrong[np.argsort(-prob[wrong].max(1))][:6]]
    samples = [
        {"id": f"T-{i:05d}", "label": int(yte[i]), "pred": int(pred[i]), "p": r(prob[i].max(), 4), "probs": [r(v, 5) for v in prob[i]], "img": b64img(xte[i]), "kind": "mistake" if i in worst else "random"}
        for i in pick + worst
    ]
    means = [b64img(np.round(xtr_all[ytr_all == dgt].mean(axis=0))) for dgt in range(10)]
    ink = [r(float((xtr_all[ytr_all == dgt] > 50).mean()), 4) for dgt in range(10)]
  7. 7

    Export the JSON

    Writes the int8 weights, model comparison, training histories, confusion matrix, digit averages and samples to src/data/models/handwritten-digits.json.

    ml/train.py:3003 · 29 lines

    save(
        "handwritten-digits",
        {
            "dataset": {
                "name": "MNIST handwritten digits",
                "source": "https://yann.lecun.com/exdb/mnist/",
                "download": "https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz",
                "license": "CC BY-SA 3.0 (Yann LeCun and Corinna Cortes)",
                "train": int(len(xtr_all)),
                "test": int(len(xte)),
                "size": 28,
            },
            "split": {"train": int(len(tr_idx)), "val": int(len(val_idx)), "test": int(len(xte))},
            "counts": {"train": np.bincount(ytr_all).tolist(), "test": np.bincount(yte).tolist()},
            "comparison": comparison,
            "chosen": chosen,

In your browser · TypeScript

  1. 8

    Unpack the 8-bit weights

    Each weight was stored as an int8 with one scale per output channel; multiplying back gives the float the network uses.

    src/lib/cnn.ts:9 · 7 lines

    export function dequant(l: QLayer): Float64Array {
      const bin = atob(l.q);
      const out = l.shape[l.shape.length - 1];
      const w = new Float64Array(bin.length);
      for (let i = 0; i < bin.length; i++) w[i] = ((bin.charCodeAt(i) << 24) >> 24) * l.scale[i % out];
      return w;
    }
  2. 9

    The CNN, layer by layer, in TypeScript

    Convolution, ReLU, pooling, dense layers and softmax, returning every intermediate activation for the neuron view.

    src/lib/cnn.ts:93 · 27 lines

    export function createDigitNet(m: DigitCnn) {
      const W1 = dequant(m.conv1);
      const W2 = dequant(m.conv2);
      const W3 = dequant(m.dense1);
      const W4 = dequant(m.out);
      const c1 = m.conv1.shape[3];
      const c2 = m.conv2.shape[3];
      const d1 = m.dense1.shape[1];
      const d2 = m.out.shape[1];
    
      /** `pixels`: 784 values in 0–1, row by row. Returns class probabilities and every intermediate activation. */
      function run(pixels: ArrayLike<number>) {
        const input: Tensor = { data: Float64Array.from(pixels), h: 28, w: 28, c: 1 };
        const a1 = conv(input, W1, m.conv1.b, c1);
        const p1 = pool(a1);
        const a2 = conv(p1, W2, m.conv2.b, c2);

Notes

The decisions behind it.

Approach

  1. 1

    Kept the official 10,000-image test set for the end; held back 5,000 training images for early stopping.

  2. 2

    Trained four networks of growing power with TensorFlow/Keras: no hidden layer, one dense hidden layer, a small CNN, and the same CNN with augmentation.

  3. 3

    Tested every model on the plain test set and on shifted, rotated and shrunk copies of it.

  4. 4

    Quantised the chosen CNN to 8 bits per weight and re-tested it with the exact forward pass the browser runs.

Test accuracy · shifted digits

Softmax regression (no hidden layer)92.74% · 33.4%
Dense network (1 hidden layer)98.06% · 47.7%
Small CNN99.21% · 71.3%
Small CNN + augmentation99.25% · 97.0%shipped

Honest note

MNIST is a solved benchmark: its digits are clean, centred and written by a few thousand people decades ago, so 99.24% on its test set says more about the method than about real-world handwriting. Digits drawn with a mouse look different, which is exactly why the shipped model was trained with augmentation; it still slips on very thin, off-centre or unusual digits. The plain CNN scores 99.21% on the test set, close to the augmented one, but falls to 71.3% on shifted digits.

Reproduce: ml/train.py → mnist()

dataset source

Next project

Hand Gestures