←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

A drawing game against a CNN: it guesses your doodle while you draw, across 30 everyday things.

89.1%

First-guess accuracy (test)

95.9%

Within three guesses

30

Things it knows

222 KB

Weights

Overview

Recognising a quick sketch is easy for people and surprisingly hard for a computer: everyone draws a cat differently, fast and messily. Google's Quick, Draw! project collected millions of 20-second doodles; this project trains a convolutional network on a sample of them and turns it into a game.

You get a word and 20 seconds. Your strokes are framed exactly like the training doodles, the network runs in your browser many times a second, and it calls the answer as soon as your word is its top guess. Four networks were compared; the shipped one guesses 89.1% of unseen test doodles first time and 95.9% within three guesses.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    300,000 doodles

    10,000 per word, 30 words

  2. 02

    Prepare

    28×28

    240,000 train · 30,000 val · 30,000 test

  3. 03

    Baseline

    62.7%

    No hidden layer

  4. 04

    CNN

    227,614 weights

    Wider CNN + augmentation

  5. 05

    Ship

    89.1%

    222 KB of 8-bit weights

  6. 06

    Play

    6 × 20 s

    Live guesses while you draw

Inside the model

Inside the doodle guesser.

The network behind the game is a small convolutional network. Scroll to follow a real test doodle through every layer, computed live with its real weights.

doodle
0552422551830021323325422200222239227220023224025221922111315323725525425525402725525512017

28 × 28 = 784 numbers, 0 (paper) to 255 (ink). That is all the network gets.

0.05-0.18-0.170.120.16-0.19-0.010.26-0.20
filter 2 of 32→

At every position: 9 pixels × 9 weights, added up. Dark where the picture matches the filter's pattern.

32 filters → 32 feature maps (26 × 26)

max-pool: 26 × 26 → 13 × 13

→

Each 2 × 2 block keeps only its strongest value: a quarter of the numbers, the same pattern, and a stroke can shift a pixel without changing much.

layer 2: 64 filters, each looking at all 32 maps at once (first 32 shown)

These combine strokes into parts: corners, loops, crossings. Then another pool: 5 × 5 × 64 = 1600 numbers.

1600 numbers → 128 neurons → 30 answers

→
applecakecoffee cupclockcatdog

Each dot is one neuron: a weighted sum of everything before it, then “keep it if positive” (ReLU). Darker = more active.

softmax: scores → chances that add up to 100%

apple
98.5%
cake
1.3%
coffee cup
0.1%
clock
0.0%
cat
0.0%

accuracy after each pass over the training data

100%59%15 EPOCHS

■ held-out validation images · ■ training images

Same filter everywhere

A stroke detector learned in one corner works in every corner.

Few weights

A filter has 9 weights per input map, however big the picture.

Layers build up

Edges → parts → whole objects, from one layer to the next.

Nothing hand-made

Every filter here was learned from examples, none designed by me.

■ blue weights subtract, ■ orange add.

The data

300,000 real doodles, 30 things.

People around the world were given a word and 20 seconds to draw it. Google published those drawings as the Quick, Draw! dataset; this network learned from a sample of 10,000 per word.

Quick, Draw! (sample)

CC BY 4.0 (Google). For each of the 30 words, 10,000 drawings taken from five places spread through Google's file (each word has over a hundred thousand), then split 8,000 / 1,000 / 1,000 for training, validation and test. The dataset was moderated by Google, which notes some drawings may still be inappropriate, so no raw training doodle is shown on this page: only averages, and your own drawings.

240,000

train

30,000

validation

30,000

test

28×28

pixels each

Quick, Draw! dataset

Column dictionary

● used by the model

ColumnTypeWhat it means
drawing28×28 pixelsA doodle drawn in under 20 seconds, cropped and scaled so its longest side fills the square
word1 of 30What the person was asked to draw

01

Your strokes

Every line you draw, recorded as points

02

Frame

Crop to the drawing, scale the longest side to 26 px, centre in 28×28

03

CNN

Two convolution blocks find lines, curves and corners; 227,614 weights in 8 bits

04

30 scores

One probability per thing it knows, recomputed while you draw

05

The duel

It wins the round once your word is its top guess with at least 25%

Play

Doodle Duel: you draw, it guesses.

Six words, twenty seconds each. Draw fast: the network guesses while you draw and calls it the moment it is confident. Works with a finger on a phone.

Live demo

runs in your browser

✏️ 🤖

Draw it. The network has to guess it.

6 rounds, 20 seconds each. It guesses while you draw, so speed beats art. It knows 30 things:

🍎 apple🍌 banana🚲 bicycle🐦 bird📖 book🦋 butterfly🎂 cake🚗 car🐱 cat🕒 clock☁️ cloud☕ coffee cup🐶 dog🐟 fish🌸 flower🎸 guitar🍔 hamburger🏠 house🍦 ice cream🔑 key💡 light bulb🌙 moon⛰️ mountain🍕 pizza🌈 rainbow⭐ star☀️ sun🌳 tree☂️ umbrella✈️ airplane

What people draw

The average of 8,000 doodles per word.

Each tile is the pixel-by-pixel average of every training drawing for that word. Shared habits survive the averaging: the round apple, the triangle roof, the arc of a rainbow.

🍎 apple

🍌 banana

🚲 bicycle

🐦 bird

📖 book

🦋 butterfly

🎂 cake

🚗 car

🐱 cat

🕒 clock

☁️ cloud

☕ coffee cup

🐶 dog

🐟 fish

🌸 flower

🎸 guitar

🍔 hamburger

🏠 house

🍦 ice cream

🔑 key

💡 light bulb

🌙 moon

⛰️ mountain

🍕 pizza

🌈 rainbow

⭐ star

☀️ sun

🌳 tree

☂️ umbrella

✈️ airplane

How good is it

89.1% of test doodles guessed first time, 95.9% within three guesses.

Four networks of growing power, tested on 30,000 doodles none of them saw. The chosen one was picked on validation accuracy, then shrunk to 8-bit weights and tested again with the exact forward pass your browser runs.

Four networks, same test doodles

No hidden layer already gets 63%; convolutions add the rest by finding strokes and shapes anywhere in the square.

ModelAccuracyTop-3Weights
Softmax regression (no hidden layer)62.7%80.4%23,550
Dense network (1 hidden layer)82.1%92.4%208,670
Small CNN88.1%95.3%58,014
Wider CNN + augmentation ✓89.1%95.9%227,614

Shipped in 8 bits: 89.08% (full precision 89.12%), 222 KB of weights.

Most confused pairs (test set)

The mistakes look like ones people would make: shapes that share an outline.

🐱 cat read as 🐶 dog116 of 1,000
🍌 banana read as 🌙 moon95 of 1,000
🐶 dog read as 🐱 cat92 of 1,000
🌙 moon read as 🍌 banana68 of 1,000
🐦 bird read as 🐶 dog51 of 1,000
🌸 flower read as 🌳 tree46 of 1,000
💡 light bulb read as ☀️ sun43 of 1,000
🌙 moon read as 🍕 pizza42 of 1,000
Accuracy for each word

Rainbow (97%) and house (96%) are easiest; bird (70%) and dog (76%) are hardest.

0%50%100%97%rainbowhousemountainbookice creamclocksunhamburgerbutterflytreebicycleumbrellaappleguitarcakestarcarfishpizzacloudflowercoffee cuplight bulbbananaairplanekeycatmoondog70%bird
What this project is, and isn't

A CNN trained from scratch on public doodles, playing a game in your browser.

Here
ModelWider CNN + augmentation: 227,614 weights trained in TensorFlow, int8 in the browser; the forward pass is TypeScript, checked against NumPy
Data240,000 training doodles from Google's Quick, Draw! dataset (CC BY 4.0); 10,000 of the 100,000+ per word
GameRounds, timer, live guesses, voice and the 28×28 'what it sees' view are built for this page; nothing you draw leaves your device
LimitsOnly 30 words; it was trained on fast doodles, so detailed drawings or unusual angles can confuse it more than a quick sketch

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

doodles() from ml/train.py + 3 helpers (r, save, QD_CATEGORIES)

pip install numpy pandas scikit-learn tensorflow · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load and split the doodles

    Seeds TensorFlow, then shuffles each category's drawings and takes 8000 for training and 1000 each for validation and test, as 28x28 images.

    ml/train.py:4032 · 26 lines

    import base64
    import time
    
    import keras
    import tensorflow as tf
    from keras import layers
    
    print("doodle-duel")
    keras.utils.set_random_seed(SEED)
    tf.config.experimental.enable_op_determinism()
    K = len(QD_CATEGORIES)
    rng = np.random.default_rng(SEED)
    parts = {"train": ([], []), "val": ([], []), "test": ([], [])}
    available = {}
    for k, c in enumerate(QD_CATEGORIES):
        a = np.load(RAW / "quickdraw" / f"{c}.npy")
  2. 2

    Define four candidate networks

    A tf.data pipeline scales pixels to 0–1 in batches of 256; candidates are softmax regression, a dense network, a small CNN and a wider CNN with augmentation.

    ml/train.py:4060 · 38 lines

    def ds(X, y, shuffle):
        d = tf.data.Dataset.from_tensor_slices((X, y))
        if shuffle:
            d = d.shuffle(len(X), seed=SEED, reshuffle_each_iteration=True)
        return d.batch(256).map(lambda a, b: (tf.cast(a, tf.float32) / 255.0, b)).prefetch(2)
    
    def predict(model, X):
        return model.predict(ds(X, np.zeros(len(X), dtype=np.int32), False), verbose=0)
    
    def softmax_reg():
        return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(K)], name="softmax")
    
    def mlp():
        return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(256, activation="relu"), layers.Dropout(0.3), layers.Dense(K)], name="mlp")
    
    def cnn(c1, c2, d1, augment):
  3. 3

    Train and compare the networks

    Each network trains up to 15 epochs with Adam, early stopping and learning-rate halving, and is scored on test accuracy and top-3 accuracy.

    ml/train.py:4099 · 18 lines

    comparison, histories, trained = [], {}, {}
    tr_ds, va_ds = ds(Xtr, ytr, True), ds(Xva, yva, False)
    for name, build in builders.items():
        keras.utils.set_random_seed(SEED)
        model = build()
        model.compile(optimizer=keras.optimizers.Adam(1e-3), loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"])
        t0 = time.time()
        h = model.fit(
            tr_ds, validation_data=va_ds, epochs=15, verbose=0,
            callbacks=[keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True, monitor="val_loss"), keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=1, min_lr=1e-5)],
        )
        secs = time.time() - t0
        Pv, Pt = predict(model, Xva), predict(model, Xte)
        top3 = float(np.mean([y in np.argsort(-p)[:3] for p, y in zip(Pt, yte)]))
        comparison.append({"model": name, "params": int(model.count_params()), "val_acc": r(float(np.mean(Pv.argmax(1) == yva)), 4), "acc": r(float(np.mean(Pt.argmax(1) == yte)), 4), "top3": r(top3, 4), "epochs": len(h.history["loss"]), "seconds": r(secs, 1)})
        histories[name] = {k: [r(v, 6 if k == "learning_rate" else 4) for v in vals] for k, vals in h.history.items()}
        trained[name] = model
        print(f"  {name:38s} params={model.count_params():7d} val={comparison[-1]['val_acc']:.4f} test={comparison[-1]['acc']:.4f} top3={top3:.4f} epochs={len(h.history['loss'])} {secs:.0f}s", flush=True)
  4. 4

    Pick a CNN, quantise and re-test

    The CNN with the best validation accuracy is kept, stored as int8 per output unit, and re-scored on the test set with a NumPy copy of the browser's src/lib/cnn.ts.

    ml/train.py:4119 · 48 lines

    cnn_names = [c_["model"] for c_ in comparison if "CNN" in c_["model"]]
    chosen = max(cnn_names, key=lambda n_: next(c_["val_acc"] for c_ in comparison if c_["model"] == n_))  # by validation, not test
    model = trained[chosen]
    
    def q8(w):
        moved = w
        scale = np.abs(moved).reshape(-1, moved.shape[-1]).max(axis=0) / 127
        scale = np.array([float(f"{s:.8g}") for s in scale])
        return np.round(moved / scale).astype(np.int8), scale
    
    Lw = {ly.name: ly.get_weights() for ly in model.layers if ly.get_weights()}
    export, deq = {}, {}
    for name in ["conv1", "conv2", "dense1", "out"]:
        w, b = Lw[name]
        q, scale = q8(w)
        export[name] = {"shape": list(w.shape), "scale": [float(s) for s in scale], "b": [r(v, 6) for v in b], "q": base64.b64encode(q.tobytes()).decode("ascii")}
  5. 5

    Analyse the mistakes

    Builds the confusion matrix, per-category accuracy and the 12 most common mix-ups, plus each category's average drawing and share of inked pixels.

    ml/train.py:4168 · 9 lines

    conf = [[int(((yte == a) & (pred == p)).sum()) for p in range(K)] for a in range(K)]
    per_class = [r(conf[k][k] / sum(conf[k]), 4) for k in range(K)]
    pairs = sorted([(conf[a][p], a, p) for a in range(K) for p in range(K) if a != p], reverse=True)[:12]
    
    def b64img(im):
        return base64.b64encode(np.asarray(im, dtype=np.uint8).reshape(-1).tobytes()).decode("ascii")
    
    means = [b64img(np.round(Xtr[ytr == k].mean(axis=0))) for k in range(K)]
    ink = [r(float((Xtr[ytr == k] > 0).mean()), 4) for k in range(K)]
  6. 6

    Save parity data and JSON

    Writes up to 500 test doodles with their logits to doodle_parity.json for checking the browser, then saves the int8 model and results to doodle-duel.json.

    ml/train.py:4178 · 28 lines

    json.dump({"imgs": [b64img(im) for im in Xte[::60][:500]], "logits": logits[::60][:500].tolist(), "y": yte[::60][:500].tolist()}, open(RAW / "doodle_parity.json", "w"))
    save(
        "doodle-duel",
        {
            "dataset": {
                "name": "Quick, Draw! (sample)",
                "source": "https://github.com/googlecreativelab/quickdraw-dataset",
                "license": "CC BY 4.0 (Google)",
                "per_class": 10000,
                "available": available,
                "size": 28,
            },
            "categories": QD_CATEGORIES,
            "split": {"train": int(len(Xtr)), "val": int(len(Xva)), "test": int(len(Xte))},
            "comparison": comparison,
            "chosen": chosen,

In your browser · TypeScript

  1. 7

    Frame your strokes like the training doodles

    Crop to the drawing, scale the longest side to 26 of 28 pixels, centre, draw with the dataset's stroke width and average down to 28×28.

    src/components/demos/doodle-demo.tsx:29 · 30 lines

    /** Frame the strokes like the Quick, Draw! bitmaps: longest side 26 px, centred in 28×28, same stroke width; area-average down. */
    export function rasterize(strokes: Stroke[]): Float64Array | null {
      const pts = strokes.flat();
      if (!pts.length) return null;
      let x0 = Infinity, y0 = Infinity, x1 = -Infinity, y1 = -Infinity;
      for (const [x, y] of pts) {
        x0 = Math.min(x0, x); y0 = Math.min(y0, y); x1 = Math.max(x1, x); y1 = Math.max(y1, y);
      }
      const s = 260 / Math.max(x1 - x0, y1 - y0, 1);
      const ox = 140 - ((x0 + x1) / 2) * s;
      const oy = 140 - ((y0 + y1) / 2) * s;
      work ??= Object.assign(document.createElement("canvas"), { width: 280, height: 280 });
      const ctx = work.getContext("2d", { willReadFrequently: true })!;
      ctx.clearRect(0, 0, 280, 280);
      ctx.lineWidth = LINE;
      ctx.lineCap = "round";
  2. 8

    The CNN, layer by layer, in TypeScript

    The same forward pass as the digits project, with 30 outputs; it runs many times a second while you draw.

    src/lib/cnn.ts:93 · 27 lines

    export function createDigitNet(m: DigitCnn) {
      const W1 = dequant(m.conv1);
      const W2 = dequant(m.conv2);
      const W3 = dequant(m.dense1);
      const W4 = dequant(m.out);
      const c1 = m.conv1.shape[3];
      const c2 = m.conv2.shape[3];
      const d1 = m.dense1.shape[1];
      const d2 = m.out.shape[1];
    
      /** `pixels`: 784 values in 0–1, row by row. Returns class probabilities and every intermediate activation. */
      function run(pixels: ArrayLike<number>) {
        const input: Tensor = { data: Float64Array.from(pixels), h: 28, w: 28, c: 1 };
        const a1 = conv(input, W1, m.conv1.b, c1);
        const p1 = pool(a1);
        const a2 = conv(p1, W2, m.conv2.b, c2);
Data download scriptml/quickdraw_data.py

ml/quickdraw_data.py:1 · 62 lines

"""
Fetch a sample of the Quick, Draw! dataset (Google, CC BY 4.0) for the Doodle Duel project.
For each category: 5 chunks of 2,000 drawings spread through the full file (HTTP range requests),
so the sample is not just the start of the file. 28x28 uint8 bitmaps -> ml/data/raw/quickdraw/<category>.npy
Run:  ml/.venv/Scripts/python -I ml/quickdraw_data.py
"""
from __future__ import annotations

import ast
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

import numpy as np

Notes

The decisions behind it.

Approach

  1. 1

    Fetched 10,000 doodles per word from five places spread through each of Google's files, then split 8,000 / 1,000 / 1,000.

  2. 2

    Measured how the dataset frames a drawing (longest side 26 of 28 pixels, centred) and framed the visitor's strokes the same way in the browser.

  3. 3

    Compared four networks in TensorFlow: no hidden layer, one dense layer, a small CNN and a wider CNN with augmentation; picked on validation accuracy.

  4. 4

    Shrank the chosen CNN to 8-bit weights, re-tested it with the browser's forward pass, and built the game around it: rounds, timer, live guesses and voice.

Test accuracy · top-3

Softmax regression (no hidden layer)62.7% · 80.4%
Dense network (1 hidden layer)82.1% · 92.4%
Small CNN88.1% · 95.3%
Wider CNN + augmentation89.1% · 95.9%shipped

Honest note

The network only knows 30 words, so anything else still gets the closest of them. It learned from 20-second doodles, so careful, detailed drawings can confuse it more than a quick sketch. The training sample is 10,000 of the 100,000+ drawings per word, taken from five places in each file rather than fully at random. Its most common mix-up is cat read as dog (116 of 1,000 test doodles).

Reproduce: ml/train.py → doodles() · ml/quickdraw_data.py

dataset source

Next project

Face Mood Mirror