Computer vision
A drawing game against a CNN: it guesses your doodle while you draw, across 30 everyday things.
89.1%
First-guess accuracy (test)
95.9%
Within three guesses
30
Things it knows
222 KB
Weights
Overview
Recognising a quick sketch is easy for people and surprisingly hard for a computer: everyone draws a cat differently, fast and messily. Google's Quick, Draw! project collected millions of 20-second doodles; this project trains a convolutional network on a sample of them and turns it into a game.
You get a word and 20 seconds. Your strokes are framed exactly like the training doodles, the network runs in your browser many times a second, and it calls the answer as soon as your word is its top guess. Four networks were compared; the shipped one guesses 89.1% of unseen test doodles first time and 95.9% within three guesses.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
300,000 doodles
10,000 per word, 30 words
- 02
Prepare
28×28
240,000 train · 30,000 val · 30,000 test
- 03
Baseline
62.7%
No hidden layer
- 04
CNN
227,614 weights
Wider CNN + augmentation
- 05
Ship
89.1%
222 KB of 8-bit weights
- 06
Play
6 × 20 s
Live guesses while you draw
Inside the model
Inside the doodle guesser.
The network behind the game is a small convolutional network. Scroll to follow a real test doodle through every layer, computed live with its real weights.
28 × 28 = 784 numbers, 0 (paper) to 255 (ink). That is all the network gets.
At every position: 9 pixels × 9 weights, added up. Dark where the picture matches the filter's pattern.
32 filters → 32 feature maps (26 × 26)
max-pool: 26 × 26 → 13 × 13
Each 2 × 2 block keeps only its strongest value: a quarter of the numbers, the same pattern, and a stroke can shift a pixel without changing much.
layer 2: 64 filters, each looking at all 32 maps at once (first 32 shown)
These combine strokes into parts: corners, loops, crossings. Then another pool: 5 × 5 × 64 = 1600 numbers.
1600 numbers → 128 neurons → 30 answers
Each dot is one neuron: a weighted sum of everything before it, then “keep it if positive” (ReLU). Darker = more active.
softmax: scores → chances that add up to 100%
accuracy after each pass over the training data
■ held-out validation images · ■ training images
Same filter everywhere
A stroke detector learned in one corner works in every corner.
Few weights
A filter has 9 weights per input map, however big the picture.
Layers build up
Edges → parts → whole objects, from one layer to the next.
Nothing hand-made
Every filter here was learned from examples, none designed by me.
■ blue weights subtract, ■ orange add.
The data
300,000 real doodles, 30 things.
People around the world were given a word and 20 seconds to draw it. Google published those drawings as the Quick, Draw! dataset; this network learned from a sample of 10,000 per word.
Quick, Draw! (sample)
CC BY 4.0 (Google). For each of the 30 words, 10,000 drawings taken from five places spread through Google's file (each word has over a hundred thousand), then split 8,000 / 1,000 / 1,000 for training, validation and test. The dataset was moderated by Google, which notes some drawings may still be inappropriate, so no raw training doodle is shown on this page: only averages, and your own drawings.
240,000
train
30,000
validation
30,000
test
28×28
pixels each
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| drawing | 28×28 pixels | A doodle drawn in under 20 seconds, cropped and scaled so its longest side fills the square |
| word | 1 of 30 | What the person was asked to draw |
01
Your strokes
Every line you draw, recorded as points
02
Frame
Crop to the drawing, scale the longest side to 26 px, centre in 28×28
03
CNN
Two convolution blocks find lines, curves and corners; 227,614 weights in 8 bits
04
30 scores
One probability per thing it knows, recomputed while you draw
05
The duel
It wins the round once your word is its top guess with at least 25%
Play
Doodle Duel: you draw, it guesses.
Six words, twenty seconds each. Draw fast: the network guesses while you draw and calls it the moment it is confident. Works with a finger on a phone.
Live demo
runs in your browser
✏️ 🤖
Draw it. The network has to guess it.
6 rounds, 20 seconds each. It guesses while you draw, so speed beats art. It knows 30 things:
What people draw
The average of 8,000 doodles per word.
Each tile is the pixel-by-pixel average of every training drawing for that word. Shared habits survive the averaging: the round apple, the triangle roof, the arc of a rainbow.
🍎 apple
🍌 banana
🚲 bicycle
🐦 bird
📖 book
🦋 butterfly
🎂 cake
🚗 car
🐱 cat
🕒 clock
☁️ cloud
☕ coffee cup
🐶 dog
🐟 fish
🌸 flower
🎸 guitar
🍔 hamburger
🏠 house
🍦 ice cream
🔑 key
💡 light bulb
🌙 moon
⛰️ mountain
🍕 pizza
🌈 rainbow
⭐ star
☀️ sun
🌳 tree
☂️ umbrella
✈️ airplane
How good is it
89.1% of test doodles guessed first time, 95.9% within three guesses.
Four networks of growing power, tested on 30,000 doodles none of them saw. The chosen one was picked on validation accuracy, then shrunk to 8-bit weights and tested again with the exact forward pass your browser runs.
No hidden layer already gets 63%; convolutions add the rest by finding strokes and shapes anywhere in the square.
| Model | Accuracy | Top-3 | Weights |
|---|---|---|---|
| Softmax regression (no hidden layer) | 62.7% | 80.4% | 23,550 |
| Dense network (1 hidden layer) | 82.1% | 92.4% | 208,670 |
| Small CNN | 88.1% | 95.3% | 58,014 |
| Wider CNN + augmentation ✓ | 89.1% | 95.9% | 227,614 |
Shipped in 8 bits: 89.08% (full precision 89.12%), 222 KB of weights.
The mistakes look like ones people would make: shapes that share an outline.
Rainbow (97%) and house (96%) are easiest; bird (70%) and dog (76%) are hardest.
A CNN trained from scratch on public doodles, playing a game in your browser.
| Here | |
|---|---|
| Model | Wider CNN + augmentation: 227,614 weights trained in TensorFlow, int8 in the browser; the forward pass is TypeScript, checked against NumPy |
| Data | 240,000 training doodles from Google's Quick, Draw! dataset (CC BY 4.0); 10,000 of the 100,000+ per word |
| Game | Rounds, timer, live guesses, voice and the 28×28 'what it sees' view are built for this page; nothing you draw leaves your device |
| Limits | Only 30 words; it was trained on fast doodles, so detailed drawings or unusual angles can confuse it more than a quick sketch |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
doodles() from ml/train.py + 3 helpers (r, save, QD_CATEGORIES)
pip install numpy pandas scikit-learn tensorflow · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load and split the doodles
Seeds TensorFlow, then shuffles each category's drawings and takes 8000 for training and 1000 each for validation and test, as 28x28 images.
ml/train.py:4032 · 26 lines
import base64 import time import keras import tensorflow as tf from keras import layers print("doodle-duel") keras.utils.set_random_seed(SEED) tf.config.experimental.enable_op_determinism() K = len(QD_CATEGORIES) rng = np.random.default_rng(SEED) parts = {"train": ([], []), "val": ([], []), "test": ([], [])} available = {} for k, c in enumerate(QD_CATEGORIES): a = np.load(RAW / "quickdraw" / f"{c}.npy") - 2
Define four candidate networks
A tf.data pipeline scales pixels to 0–1 in batches of 256; candidates are softmax regression, a dense network, a small CNN and a wider CNN with augmentation.
ml/train.py:4060 · 38 lines
def ds(X, y, shuffle): d = tf.data.Dataset.from_tensor_slices((X, y)) if shuffle: d = d.shuffle(len(X), seed=SEED, reshuffle_each_iteration=True) return d.batch(256).map(lambda a, b: (tf.cast(a, tf.float32) / 255.0, b)).prefetch(2) def predict(model, X): return model.predict(ds(X, np.zeros(len(X), dtype=np.int32), False), verbose=0) def softmax_reg(): return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(K)], name="softmax") def mlp(): return keras.Sequential([keras.Input((28, 28, 1)), layers.Flatten(), layers.Dense(256, activation="relu"), layers.Dropout(0.3), layers.Dense(K)], name="mlp") def cnn(c1, c2, d1, augment): - 3
Train and compare the networks
Each network trains up to 15 epochs with Adam, early stopping and learning-rate halving, and is scored on test accuracy and top-3 accuracy.
ml/train.py:4099 · 18 lines
comparison, histories, trained = [], {}, {} tr_ds, va_ds = ds(Xtr, ytr, True), ds(Xva, yva, False) for name, build in builders.items(): keras.utils.set_random_seed(SEED) model = build() model.compile(optimizer=keras.optimizers.Adam(1e-3), loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"]) t0 = time.time() h = model.fit( tr_ds, validation_data=va_ds, epochs=15, verbose=0, callbacks=[keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True, monitor="val_loss"), keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=1, min_lr=1e-5)], ) secs = time.time() - t0 Pv, Pt = predict(model, Xva), predict(model, Xte) top3 = float(np.mean([y in np.argsort(-p)[:3] for p, y in zip(Pt, yte)])) comparison.append({"model": name, "params": int(model.count_params()), "val_acc": r(float(np.mean(Pv.argmax(1) == yva)), 4), "acc": r(float(np.mean(Pt.argmax(1) == yte)), 4), "top3": r(top3, 4), "epochs": len(h.history["loss"]), "seconds": r(secs, 1)}) histories[name] = {k: [r(v, 6 if k == "learning_rate" else 4) for v in vals] for k, vals in h.history.items()} trained[name] = model print(f" {name:38s} params={model.count_params():7d} val={comparison[-1]['val_acc']:.4f} test={comparison[-1]['acc']:.4f} top3={top3:.4f} epochs={len(h.history['loss'])} {secs:.0f}s", flush=True) - 4
Pick a CNN, quantise and re-test
The CNN with the best validation accuracy is kept, stored as int8 per output unit, and re-scored on the test set with a NumPy copy of the browser's src/lib/cnn.ts.
ml/train.py:4119 · 48 lines
cnn_names = [c_["model"] for c_ in comparison if "CNN" in c_["model"]] chosen = max(cnn_names, key=lambda n_: next(c_["val_acc"] for c_ in comparison if c_["model"] == n_)) # by validation, not test model = trained[chosen] def q8(w): moved = w scale = np.abs(moved).reshape(-1, moved.shape[-1]).max(axis=0) / 127 scale = np.array([float(f"{s:.8g}") for s in scale]) return np.round(moved / scale).astype(np.int8), scale Lw = {ly.name: ly.get_weights() for ly in model.layers if ly.get_weights()} export, deq = {}, {} for name in ["conv1", "conv2", "dense1", "out"]: w, b = Lw[name] q, scale = q8(w) export[name] = {"shape": list(w.shape), "scale": [float(s) for s in scale], "b": [r(v, 6) for v in b], "q": base64.b64encode(q.tobytes()).decode("ascii")} - 5
Analyse the mistakes
Builds the confusion matrix, per-category accuracy and the 12 most common mix-ups, plus each category's average drawing and share of inked pixels.
ml/train.py:4168 · 9 lines
conf = [[int(((yte == a) & (pred == p)).sum()) for p in range(K)] for a in range(K)] per_class = [r(conf[k][k] / sum(conf[k]), 4) for k in range(K)] pairs = sorted([(conf[a][p], a, p) for a in range(K) for p in range(K) if a != p], reverse=True)[:12] def b64img(im): return base64.b64encode(np.asarray(im, dtype=np.uint8).reshape(-1).tobytes()).decode("ascii") means = [b64img(np.round(Xtr[ytr == k].mean(axis=0))) for k in range(K)] ink = [r(float((Xtr[ytr == k] > 0).mean()), 4) for k in range(K)] - 6
Save parity data and JSON
Writes up to 500 test doodles with their logits to doodle_parity.json for checking the browser, then saves the int8 model and results to doodle-duel.json.
ml/train.py:4178 · 28 lines
json.dump({"imgs": [b64img(im) for im in Xte[::60][:500]], "logits": logits[::60][:500].tolist(), "y": yte[::60][:500].tolist()}, open(RAW / "doodle_parity.json", "w")) save( "doodle-duel", { "dataset": { "name": "Quick, Draw! (sample)", "source": "https://github.com/googlecreativelab/quickdraw-dataset", "license": "CC BY 4.0 (Google)", "per_class": 10000, "available": available, "size": 28, }, "categories": QD_CATEGORIES, "split": {"train": int(len(Xtr)), "val": int(len(Xva)), "test": int(len(Xte))}, "comparison": comparison, "chosen": chosen,
In your browser · TypeScript
- 7
Frame your strokes like the training doodles
Crop to the drawing, scale the longest side to 26 of 28 pixels, centre, draw with the dataset's stroke width and average down to 28×28.
src/components/demos/doodle-demo.tsx:29 · 30 lines
/** Frame the strokes like the Quick, Draw! bitmaps: longest side 26 px, centred in 28×28, same stroke width; area-average down. */ export function rasterize(strokes: Stroke[]): Float64Array | null { const pts = strokes.flat(); if (!pts.length) return null; let x0 = Infinity, y0 = Infinity, x1 = -Infinity, y1 = -Infinity; for (const [x, y] of pts) { x0 = Math.min(x0, x); y0 = Math.min(y0, y); x1 = Math.max(x1, x); y1 = Math.max(y1, y); } const s = 260 / Math.max(x1 - x0, y1 - y0, 1); const ox = 140 - ((x0 + x1) / 2) * s; const oy = 140 - ((y0 + y1) / 2) * s; work ??= Object.assign(document.createElement("canvas"), { width: 280, height: 280 }); const ctx = work.getContext("2d", { willReadFrequently: true })!; ctx.clearRect(0, 0, 280, 280); ctx.lineWidth = LINE; ctx.lineCap = "round"; - 8
The CNN, layer by layer, in TypeScript
The same forward pass as the digits project, with 30 outputs; it runs many times a second while you draw.
src/lib/cnn.ts:93 · 27 lines
export function createDigitNet(m: DigitCnn) { const W1 = dequant(m.conv1); const W2 = dequant(m.conv2); const W3 = dequant(m.dense1); const W4 = dequant(m.out); const c1 = m.conv1.shape[3]; const c2 = m.conv2.shape[3]; const d1 = m.dense1.shape[1]; const d2 = m.out.shape[1]; /** `pixels`: 784 values in 0–1, row by row. Returns class probabilities and every intermediate activation. */ function run(pixels: ArrayLike<number>) { const input: Tensor = { data: Float64Array.from(pixels), h: 28, w: 28, c: 1 }; const a1 = conv(input, W1, m.conv1.b, c1); const p1 = pool(a1); const a2 = conv(p1, W2, m.conv2.b, c2);
Data download scriptml/quickdraw_data.py
ml/quickdraw_data.py:1 · 62 lines
"""
Fetch a sample of the Quick, Draw! dataset (Google, CC BY 4.0) for the Doodle Duel project.
For each category: 5 chunks of 2,000 drawings spread through the full file (HTTP range requests),
so the sample is not just the start of the file. 28x28 uint8 bitmaps -> ml/data/raw/quickdraw/<category>.npy
Run: ml/.venv/Scripts/python -I ml/quickdraw_data.py
"""
from __future__ import annotations
import ast
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import numpy as np
Notes
The decisions behind it.
Approach
- 1
Fetched 10,000 doodles per word from five places spread through each of Google's files, then split 8,000 / 1,000 / 1,000.
- 2
Measured how the dataset frames a drawing (longest side 26 of 28 pixels, centred) and framed the visitor's strokes the same way in the browser.
- 3
Compared four networks in TensorFlow: no hidden layer, one dense layer, a small CNN and a wider CNN with augmentation; picked on validation accuracy.
- 4
Shrank the chosen CNN to 8-bit weights, re-tested it with the browser's forward pass, and built the game around it: rounds, timer, live guesses and voice.
Test accuracy · top-3
Honest note
The network only knows 30 words, so anything else still gets the closest of them. It learned from 20-second doodles, so careful, detailed drawings can confuse it more than a quick sketch. The training sample is 10,000 of the 100,000+ drawings per word, taken from five places in each file rather than fully at random. Its most common mix-up is cat read as dog (116 of 1,000 test doodles).
Reproduce: ml/train.py → doodles() · ml/quickdraw_data.py
dataset sourceNext project
Face Mood Mirror