Computer vision
Hand gestures read live from your webcam, in the browser, and a CNN experiment that shows why hand landmarks beat raw pixels.
7
Live gestures
21
Hand points tracked
8 MB, once
Model download
79.3%
Pixel CNN, unseen backgrounds
Overview
Gesture control has to work in a real room: any background, any light, any hand. The robust way is to find the hand first and describe it by its joints, not by its pixels. MediaPipe does exactly that, tracking 21 points per hand in real time, and its gesture model reads fist, palm, victory and more from those points.
The live demo runs MediaPipe in your browser with WebAssembly and the GPU, smooths its answers over the last frames and turns them into a rock-paper-scissors game. Below it, my own CNN trained on rendered hands shows the alternative: on backgrounds it never saw it falls to 48.7%, and background swaps only lift it to 79.3%.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Camera
your webcam
Frames stay on the device
- 02
Hand points
21 landmarks
MediaPipe's tracker, WebAssembly + GPU
- 03
Gesture
7 classes
MediaPipe's classifier on the landmarks
- 04
Smoothing
10 frames
Majority vote, so one noisy frame can't flip it
- 05
Game
✊ ✋ ✌️
Fist, palm and victory become your move
- 06
Experiment
48.7% → 79.3%
My pixel CNN on unseen backgrounds
Inside the model
From a hand to a gesture.
The live demo stands on Google's pretrained hand networks; my own CNN on pixels is the experiment that shows why. Scroll through both.
The data
A hand tracker for the camera, a dataset for the experiment.
The live demo uses MediaPipe's gesture recognizer, trained by Google on real hands. To show why that approach works, I also trained my own CNN on the Rock, Paper, Scissors images.
01
Camera frame
640×480 from your webcam, kept on the device
02
Find the hand
MediaPipe's palm detector locates it, then tracks it frame to frame
03
21 hand points
Knuckles and fingertips in 3D: shape without background
04
Gesture
A small classifier reads the points: fist, palm, victory…
05
Steady answer
Majority vote over the last 10 frames
Rock, Paper, Scissors (Laurence Moroney, 2019)
For the CNN experiment: 2,520 training and 372 test images of rendered hands, 300×300 pixels on a white background, with different skin tones, ages and genders. Licence: CC BY 2.0, attribution to Laurence Moroney.
2,520
training images
372
test images
840
per gesture
64×64
CNN input
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| image | 300×300 RGB | A rendered hand on a white background; resized to 64×64 for the CNN experiment |
| label | rock · paper · scissors | The gesture the CNN predicts |
Try it
Make a fist at your camera.
Hand tracking and gesture recognition run live in your browser on WebAssembly and your GPU. Then play rock, paper, scissors against the computer.
Live demo
runs in your browser
Turn on your camera and make a fist, an open palm or a victory sign.
The video is processed on your device and never uploaded or recorded. Sample hands: Laurence Moroney, CC BY 2.0.
Gesture
Waiting
✊
Closed fist
✋
Open palm
✌️
Victory
👍
Thumb up
👎
Thumb down
☝️
Pointing up
🤟
I love you
Rock, paper, scissors
you 0 · cpu 0 · draw 0
Hold ✊, ✋ or ✌️ at the camera and press play. Your move is read on “shoot”.
Why landmarks, not pixels
A CNN on pixels learns the background too.
The dataset's hands are all on white. A network trained on those pixels can learn 'hand = whatever isn't white': fine on the test set, wrong on a real room. So I trained a small CNN, added augmentation, then added copies with the white swapped for random textures, and tested every model on backgrounds it never saw.
- Original test images· white background, like training
- Rotated ±30°· hands held at an angle
- Darker· 55% brightness, a dim room
- New background· unseen colour textures behind the hand
840 training and 124 test images of each, so accuracy can't be inflated by guessing one class.
✊
Rock
840 + 124
✋
Paper
840 + 124
✌️
Scissors
840 + 124
Live demo
runs in your browser
My CNN, running in your browser: pick a test image, swap in a busy background, rotate or darken it, and watch the orange heatmap show where it looks.
Pick a test image
Mirror
Busy background
What it sees · 64×64
Show where it looks
The network says
✊ Rock
100.0% sure
Test image, really rock. Read correctly.
First layer: 16 maps, 64×64
Edges and skin against background.
Last layer: 64 maps, 8×8
Each one fires on a part: fingers spread, a closed fist, two fingers up. The heatmap is these maps weighted by how much each votes for the answer.
How well it works
48.7% → 79.3% on backgrounds it never saw.
Swapping backgrounds while training rescues the pixel CNN, but only partly. Hand landmarks sidestep the problem: 21 points carry the shape of the hand and nothing about the wall behind it, which is why the live demo is built on them.
Pixels alone: 63.2%. A small CNN: 95.2% on clean images. Augmentation and background swaps mostly buy robustness; clean accuracy moves by about a point. MobileNetV2 reaches 91.4% on new backgrounds with 2,261,827 parameters, 37× the CNN.
Stopped after 27 epochs. Validation uses plain white-background images; training images include swapped backgrounds and random changes, so training accuracy runs lower.
15 mistakes in 372 original test images for the 8-bit browser version.
| true ↓ · read → | ✊ | ✋ | ✌️ |
|---|---|---|---|
| ✊ Rock | 124 | 0 | 0 |
| ✋ Paper | 11 | 109 | 4 |
| ✌️ Scissors | 0 | 0 | 124 |
Seconds are CPU training time on a laptop. MobileNetV2 reuses ImageNet features and only trains its last layer; it stays a reference, the browser runs the small CNN.
| Model | Parameters | Original | Rotated | Darker | New background | Epochs | Seconds |
|---|---|---|---|---|---|---|---|
| Logistic regression on pixels | 3,075 | 63.2% | 47.9% | 63.4% | 30.6% | – | 8.7 |
| Small CNN | 60,707 | 95.2% | 87.4% | 82.0% | 47.0% | 35 | 644.4 |
| Small CNN + augmentation | 60,707 | 96.5% | 94.6% | 96.8% | 48.7% | 28 | 572.6 |
| Small CNN + augmentation + new backgrounds (in the demo) | 60,707 | 96.0% | 96.2% | 95.2% | 79.3% | 27 | 295 |
| MobileNetV2 transfer (ImageNet) + new backgrounds | 2,261,827 | 98.4% | 99.2% | 94.3% | 91.4% | 20 | 366.5 |
| In the demo, 8-bit weights | 60,707 | 96.0% | 96.2% | 95.2% | 79.0% | – | – |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
rps() from ml/train.py + 2 helpers (r, save)
pip install numpy pandas scikit-learn scipy tensorflow · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load the photos at two sizes
Seeds TensorFlow, defines texture and background-swap helpers, and loads every training and test photo at 64 px for the small CNN and 96 px for MobileNetV2.
ml/train.py:3039 · 55 lines
import base64 import os import time import keras import tensorflow as tf from keras import layers from scipy import ndimage print("hand-gestures") keras.utils.set_random_seed(SEED) tf.config.experimental.enable_op_determinism() CLASSES = ["rock", "paper", "scissors"] S, SM = 64, 96 # small CNN input, MobileNetV2 input def textures(n, seed): - 2
Make background and stress copies
Pastes busy textures behind every training hand, then makes test copies rotated up to 30 degrees, darkened, or on unseen textures to measure robustness.
ml/train.py:3096 · 28 lines
# training copies with swapped backgrounds (seed A); test stress set uses different textures (seed B) tex_tr = textures(60, SEED + 10) tex_te = textures(40, SEED + 20) g = np.random.default_rng(SEED + 11) bg_tr, bg_trm = [], [] for k, c in enumerate(CLASSES): for fn in sorted(os.listdir(tr_root / c)): img = tf.io.decode_png(tf.io.read_file(str(tr_root / c / fn)), channels=3).numpy() sw = swap_bg(img, tex_tr[g.integers(len(tex_tr))]) bg_tr.append(np.round(small(sw, S)).astype(np.uint8)) bg_trm.append(np.round(small(sw, SM)).astype(np.uint8)) bg_tr, bg_trm = np.stack(bg_tr), np.stack(bg_trm) # stress versions of the test set, built from the 300×300 originals rr = np.random.default_rng(SEED + 21) stress = {"Original": (Xte, Xtem)} - 3
Split and define the networks
Holds out 15% of training photos for validation, then defines a four-conv small CNN with optional augmentation and a frozen ImageNet MobileNetV2 with a new head.
ml/train.py:3126 · 54 lines
perm = np.random.default_rng(SEED).permutation(len(ytr_all)) nval = int(0.15 * len(perm)) va, tr = perm[:nval], perm[nval:] f32 = lambda a: a.astype("float32") / 255.0 # noqa: E731 def cnn(name, augment): aug = ( [ layers.RandomFlip("horizontal"), layers.RandomRotation(0.08, fill_mode="constant", fill_value=1.0), layers.RandomZoom(0.15, fill_mode="constant", fill_value=1.0), layers.RandomTranslation(0.1, 0.1, fill_mode="constant", fill_value=1.0), layers.RandomBrightness(0.25, value_range=(0.0, 1.0)), layers.RandomContrast(0.25), ] if augment - 4
Train and compare the models
A pixel logistic-regression baseline, three small-CNN variants and MobileNetV2 are each scored on the original test photos and every stressed copy.
ml/train.py:3182 · 36 lines
comparison, histories, trained = [], {}, {} # 0. logistic regression on 32×32 grayscale pixels def gray32(a): return tf.image.resize(a.astype("float32"), (32, 32), method="area").numpy().mean(axis=3).reshape(len(a), -1) / 255.0 t0 = time.time() logit = LogisticRegression(max_iter=3000).fit(gray32(Xtr_all[tr]), ytr_all[tr]) rob = {k: r(float(np.mean(logit.predict(gray32(v[0])) == yte)), 4) for k, v in stress.items()} comparison.append({"model": "Logistic regression on pixels", "params": int(logit.coef_.size + logit.intercept_.size), "acc": rob["Original"], "robust": rob, "epochs": 0, "seconds": r(time.time() - t0, 1)}) variants = { "Small CNN": dict(augment=False, bg=False), "Small CNN + augmentation": dict(augment=True, bg=False), "Small CNN + augmentation + new backgrounds": dict(augment=True, bg=True), } - 5
Quantise to int8 and re-test
Keeps the augmented, background-trained CNN, stores its weights as int8 per output channel and re-scores every test set with the NumPy forward pass the browser mirrors.
ml/train.py:3220 · 49 lines
chosen = "Small CNN + augmentation + new backgrounds" model = trained[chosen] # export: int8 weights per output channel; evaluate with the browser's forward pass (same padding) def q8(w): scale = np.abs(w).reshape(-1, w.shape[-1]).max(axis=0) / 127 scale = np.array([float(f"{s:.8g}") for s in scale]) return np.round(w / scale).astype(np.int8), scale L = {ly.name: ly.get_weights() for ly in model.layers if ly.get_weights()} export, deq = {}, {} for name in ["conv1", "conv2", "conv3", "conv4", "out"]: w, b = L[name] q, scale = q8(w) export[name] = {"shape": list(w.shape), "scale": [float(s) for s in scale], "b": [r(x, 6) for x in b], "q": base64.b64encode(q.tobytes()).decode("ascii")} deq[name] = (q.astype(np.float64) * scale, np.round(b.astype(np.float64), 6)) - 6
Pick demo photos
Three random test photos per gesture, up to three mistakes, and one new-background copy per gesture, each with the shipped model's probabilities.
ml/train.py:3271 · 17 lines
# samples: 3 per gesture (random) + misclassified ones, plus one swapped-background copy per gesture def b64(a): return base64.b64encode(np.asarray(a, dtype=np.uint8).tobytes()).decode("ascii") rs = np.random.default_rng(SEED + 3) pick = [int(i) for k in range(3) for i in rs.choice(np.where(yte == k)[0], size=3, replace=False)] wrong = [int(i) for i in np.where(pred != yte)[0][:3]] samples = [] for i in pick + [w for w in wrong if w not in pick]: samples.append({"id": f"T-{i:03d}", "label": int(yte[i]), "pred": int(pred[i]), "probs": [r(x, 5) for x in prob[i]], "img": b64(Xte[i]), "kind": "mistake" if pred[i] != yte[i] else "random", "variant": "original"}) bgX = stress["New background"][0] bg_logits, _ = ship_eval(bgX) bg_prob = np.exp(bg_logits - bg_logits.max(1, keepdims=True)) bg_prob /= bg_prob.sum(1, keepdims=True) for k in range(3): i = int(rs.choice(np.where(yte == k)[0])) samples.append({"id": f"B-{i:03d}", "label": int(yte[i]), "pred": int(bg_prob[i].argmax()), "probs": [r(x, 5) for x in bg_prob[i]], "img": b64(bgX[i]), "kind": "background", "variant": "new background"}) - 7
Export the JSON
Writes the int8 weights, model comparison, training history, stress results, confusion matrix and samples to src/data/models/hand-gestures.json.
ml/train.py:3290 · 26 lines
save( "hand-gestures", { "dataset": { "name": "Rock, Paper, Scissors (Laurence Moroney, 2019)", "source": "https://laurencemoroney.com/datasets.html", "download": "https://storage.googleapis.com/download.tensorflow.org/data/rps.zip", "license": "CC BY 2.0", "train": int(len(ytr_all)), "test": int(len(yte)), "size": 300, }, "classes": CLASSES, "input": S, "split": {"train": int(len(tr)), "val": int(len(va)), "test": int(len(yte))}, "comparison": comparison,
In your browser · TypeScript
- 8
Start MediaPipe on the GPU, fall back to the CPU
Tries the GPU delegate with a timeout and quietly retries on the CPU, so the demo starts on any device.
src/lib/mediapipe.ts:17 · 9 lines
export async function withGpuFallback<T>(make: (delegate: "GPU" | "CPU") => Promise<T>, timeoutMs = 8000): Promise<T> { // some browsers stall on GPU setup (blocked WebGL, background tabs): give it a few seconds, then use the CPU const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("GPU setup timed out")), timeoutMs)); try { return await Promise.race([make("GPU"), timeout]); } catch { return await make("CPU"); } } - 9
Smooth the gesture over the last frames
A majority vote over recent frames, so one noisy frame can't flip rock into paper.
src/components/demos/camera-gesture-demo.tsx:77 · 9 lines
/** Most frequent gesture over the last frames, so a single noisy frame doesn't flip the answer. */ function stable(history: string[]) { const counts = new Map<string, number>(); for (const g of history) counts.set(g, (counts.get(g) ?? 0) + 1); let best = ""; let n = 0; for (const [g, c] of counts) if (c > n) [best, n] = [g, c]; return n >= Math.ceil(history.length * 0.6) ? best : ""; } - 10
My own CNN, with a heatmap of where it looks
Same-padding convolutions, global average pooling and a class activation map for the experiment below the game.
src/lib/cnn.ts:137 · 36 lines
export function createGestureNet(m: GestureCnn) { const W = { c1: dequant(m.conv1), c2: dequant(m.conv2), c3: dequant(m.conv3), c4: dequant(m.conv4), out: dequant(m.out) }; const ch = [m.conv1.shape[3], m.conv2.shape[3], m.conv3.shape[3], m.conv4.shape[3]]; const K = m.out.shape[1]; /** `pixels`: size×size×3 values in 0–1 (RGB, row by row). Returns probabilities, feature maps and class activation maps. */ return function run(pixels: ArrayLike<number>, size: number) { const input: Tensor = { data: Float64Array.from(pixels), h: size, w: size, c: 3 }; const a1 = convSame(input, W.c1, m.conv1.b, ch[0]); const a2 = convSame(pool(a1), W.c2, m.conv2.b, ch[1]); const a3 = convSame(pool(a2), W.c3, m.conv3.b, ch[2]); const a4 = convSame(pool(a3), W.c4, m.conv4.b, ch[3]); const n = a4.h * a4.w; const gap = new Float64Array(ch[3]); for (let c = 0; c < ch[3]; c++) { let s = 0;
Notes
The decisions behind it.
Approach
- 1
Ran MediaPipe's gesture recognizer (hand detector, 21-point landmark model, gesture classifier) in the browser, self-hosted, GPU first with a CPU fallback.
- 2
Smoothed the per-frame gesture with a majority vote over the last 10 frames and mapped fist, palm and victory to a rock-paper-scissors game.
- 3
For comparison, trained my own CNN on 2,520 rendered hands with TensorFlow: plain, with augmentation, and with background swaps, plus MobileNetV2 transfer learning.
- 4
Stress-tested every CNN on rotated, darker and new-background copies of the test set; the CNN runs in the browser too, with heatmaps of where it looks.
CNN experiment: original test set · unseen backgrounds
Honest note
The live recognizer is MediaPipe's pretrained model from Google, used as it is: I built the camera pipeline, the smoothing, the game and the comparison, not the gesture model itself. It needs a decent camera and light, and struggles with very fast motion, gloves or a hand cut off by the frame. The CNN experiment uses rendered hands only, so even its stress test is synthetic; MobileNetV2 is sturdier there (91.4% on new backgrounds) but has 37× the parameters.
Reproduce: ml/train.py → rps()
dataset sourceNext project
Live Object Detection