←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

Hand gestures read live from your webcam, in the browser, and a CNN experiment that shows why hand landmarks beat raw pixels.

7

Live gestures

21

Hand points tracked

8 MB, once

Model download

79.3%

Pixel CNN, unseen backgrounds

Overview

Gesture control has to work in a real room: any background, any light, any hand. The robust way is to find the hand first and describe it by its joints, not by its pixels. MediaPipe does exactly that, tracking 21 points per hand in real time, and its gesture model reads fist, palm, victory and more from those points.

The live demo runs MediaPipe in your browser with WebAssembly and the GPU, smooths its answers over the last frames and turns them into a rock-paper-scissors game. Below it, my own CNN trained on rendered hands shows the alternative: on backgrounds it never saw it falls to 48.7%, and background swaps only lift it to 79.3%.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Camera

    your webcam

    Frames stay on the device

  2. 02

    Hand points

    21 landmarks

    MediaPipe's tracker, WebAssembly + GPU

  3. 03

    Gesture

    7 classes

    MediaPipe's classifier on the landmarks

  4. 04

    Smoothing

    10 frames

    Majority vote, so one noisy frame can't flip it

  5. 05

    Game

    ✊ ✋ ✌️

    Fist, palm and victory become your move

  6. 06

    Experiment

    48.7% → 79.3%

    My pixel CNN on unseen backgrounds

Inside the model

From a hand to a gesture.

The live demo stands on Google's pretrained hand networks; my own CNN on pixels is the experiment that shows why. Scroll through both.

Hand
CAMERA FRAMEPALM DETECTOR: A HAND IS HERE

The data

A hand tracker for the camera, a dataset for the experiment.

The live demo uses MediaPipe's gesture recognizer, trained by Google on real hands. To show why that approach works, I also trained my own CNN on the Rock, Paper, Scissors images.

01

Camera frame

640×480 from your webcam, kept on the device

02

Find the hand

MediaPipe's palm detector locates it, then tracks it frame to frame

03

21 hand points

Knuckles and fingertips in 3D: shape without background

04

Gesture

A small classifier reads the points: fist, palm, victory…

05

Steady answer

Majority vote over the last 10 frames

Rock, Paper, Scissors (Laurence Moroney, 2019)

For the CNN experiment: 2,520 training and 372 test images of rendered hands, 300×300 pixels on a white background, with different skin tones, ages and genders. Licence: CC BY 2.0, attribution to Laurence Moroney.

2,520

training images

372

test images

840

per gesture

64×64

CNN input

Laurence Moroney's datasets page

Column dictionary

● used by the model

ColumnTypeWhat it means
image300×300 RGBA rendered hand on a white background; resized to 64×64 for the CNN experiment
labelrock · paper · scissorsThe gesture the CNN predicts

Try it

Make a fist at your camera.

Hand tracking and gesture recognition run live in your browser on WebAssembly and your GPU. Then play rock, paper, scissors against the computer.

Live demo

runs in your browser

Turn on your camera and make a fist, an open palm or a victory sign.

or try a sample:

The video is processed on your device and never uploaded or recorded. Sample hands: Laurence Moroney, CC BY 2.0.

Gesture

·

Waiting

✊

Closed fist

✋

Open palm

✌️

Victory

👍

Thumb up

👎

Thumb down

☝️

Pointing up

🤟

I love you

Rock, paper, scissors

you 0 · cpu 0 · draw 0

Hold ✊, ✋ or ✌️ at the camera and press play. Your move is read on “shoot”.

Why landmarks, not pixels

A CNN on pixels learns the background too.

The experiment

The dataset's hands are all on white. A network trained on those pixels can learn 'hand = whatever isn't white': fine on the test set, wrong on a real room. So I trained a small CNN, added augmentation, then added copies with the white swapped for random textures, and tested every model on backgrounds it never saw.

  • Original test images· white background, like training
  • Rotated ±30°· hands held at an angle
  • Darker· 55% brightness, a dim room
  • New background· unseen colour textures behind the hand
Three gestures, equal shares

840 training and 124 test images of each, so accuracy can't be inflated by guessing one class.

✊

Rock

840 + 124

✋

Paper

840 + 124

✌️

Scissors

840 + 124

Live demo

runs in your browser

My CNN, running in your browser: pick a test image, swap in a busy background, rotate or darken it, and watch the orange heatmap show where it looks.

Pick a test image

stays in your browser · blue = new-background test

Mirror

Busy background

What it sees · 64×64

Show where it looks

The network says

✊ Rock

100.0% sure

Rock
100.0%
Paper
0.0%
Scissors
0.0%

Test image, really rock. Read correctly.

First layer: 16 maps, 64×64

Edges and skin against background.

Last layer: 64 maps, 8×8

Each one fires on a part: fingers spread, a closed fist, two fingers up. The heatmap is these maps weighted by how much each votes for the answer.

How well it works

48.7% → 79.3% on backgrounds it never saw.

Swapping backgrounds while training rescues the pixel CNN, but only partly. Hand landmarks sidestep the problem: 21 points carry the shape of the hand and nothing about the wall behind it, which is why the live demo is built on them.

Five models, four kinds of test

Pixels alone: 63.2%. A small CNN: 95.2% on clean images. Augmentation and background swaps mostly buy robustness; clean accuracy moves by about a point. MobileNetV2 reaches 91.4% on new backgrounds with 2,261,827 parameters, 37× the CNN.

OriginalRotated ±30°DarkerNew background
0%50%100%63%48%63%31%Pixels95%87%82%47%CNN97%95%97%49%CNN + aug.96%96%95%79%+ backgrounds98%99%94%91%MobileNetV2
Training the CNN

Stopped after 27 epochs. Validation uses plain white-background images; training images include swapped backgrounds and random changes, so training accuracy runs lower.

Training accuracyValidation accuracyWithout background swaps (val)
30%65%100%18152128Epoch
Which gestures the CNN confuses

15 mistakes in 372 original test images for the 8-bit browser version.

true ↓ · read →✊✋✌️
✊ Rock12400
✋ Paper111094
✌️ Scissors00124
All CNN models

Seconds are CPU training time on a laptop. MobileNetV2 reuses ImageNet features and only trains its last layer; it stays a reference, the browser runs the small CNN.

ModelParametersOriginalRotatedDarkerNew backgroundEpochsSeconds
Logistic regression on pixels3,07563.2%47.9%63.4%30.6%–8.7
Small CNN60,70795.2%87.4%82.0%47.0%35644.4
Small CNN + augmentation60,70796.5%94.6%96.8%48.7%28572.6
Small CNN + augmentation + new backgrounds (in the demo)60,70796.0%96.2%95.2%79.3%27295
MobileNetV2 transfer (ImageNet) + new backgrounds2,261,82798.4%99.2%94.3%91.4%20366.5
In the demo, 8-bit weights60,70796.0%96.2%95.2%79.0%––

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

rps() from ml/train.py + 2 helpers (r, save)

pip install numpy pandas scikit-learn scipy tensorflow · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load the photos at two sizes

    Seeds TensorFlow, defines texture and background-swap helpers, and loads every training and test photo at 64 px for the small CNN and 96 px for MobileNetV2.

    ml/train.py:3039 · 55 lines

    import base64
    import os
    import time
    
    import keras
    import tensorflow as tf
    from keras import layers
    from scipy import ndimage
    
    print("hand-gestures")
    keras.utils.set_random_seed(SEED)
    tf.config.experimental.enable_op_determinism()
    CLASSES = ["rock", "paper", "scissors"]
    S, SM = 64, 96  # small CNN input, MobileNetV2 input
    
    def textures(n, seed):
  2. 2

    Make background and stress copies

    Pastes busy textures behind every training hand, then makes test copies rotated up to 30 degrees, darkened, or on unseen textures to measure robustness.

    ml/train.py:3096 · 28 lines

    # training copies with swapped backgrounds (seed A); test stress set uses different textures (seed B)
    tex_tr = textures(60, SEED + 10)
    tex_te = textures(40, SEED + 20)
    g = np.random.default_rng(SEED + 11)
    bg_tr, bg_trm = [], []
    for k, c in enumerate(CLASSES):
        for fn in sorted(os.listdir(tr_root / c)):
            img = tf.io.decode_png(tf.io.read_file(str(tr_root / c / fn)), channels=3).numpy()
            sw = swap_bg(img, tex_tr[g.integers(len(tex_tr))])
            bg_tr.append(np.round(small(sw, S)).astype(np.uint8))
            bg_trm.append(np.round(small(sw, SM)).astype(np.uint8))
    bg_tr, bg_trm = np.stack(bg_tr), np.stack(bg_trm)
    
    # stress versions of the test set, built from the 300×300 originals
    rr = np.random.default_rng(SEED + 21)
    stress = {"Original": (Xte, Xtem)}
  3. 3

    Split and define the networks

    Holds out 15% of training photos for validation, then defines a four-conv small CNN with optional augmentation and a frozen ImageNet MobileNetV2 with a new head.

    ml/train.py:3126 · 54 lines

    perm = np.random.default_rng(SEED).permutation(len(ytr_all))
    nval = int(0.15 * len(perm))
    va, tr = perm[:nval], perm[nval:]
    f32 = lambda a: a.astype("float32") / 255.0  # noqa: E731
    
    def cnn(name, augment):
        aug = (
            [
                layers.RandomFlip("horizontal"),
                layers.RandomRotation(0.08, fill_mode="constant", fill_value=1.0),
                layers.RandomZoom(0.15, fill_mode="constant", fill_value=1.0),
                layers.RandomTranslation(0.1, 0.1, fill_mode="constant", fill_value=1.0),
                layers.RandomBrightness(0.25, value_range=(0.0, 1.0)),
                layers.RandomContrast(0.25),
            ]
            if augment
  4. 4

    Train and compare the models

    A pixel logistic-regression baseline, three small-CNN variants and MobileNetV2 are each scored on the original test photos and every stressed copy.

    ml/train.py:3182 · 36 lines

    comparison, histories, trained = [], {}, {}
    
    # 0. logistic regression on 32×32 grayscale pixels
    def gray32(a):
        return tf.image.resize(a.astype("float32"), (32, 32), method="area").numpy().mean(axis=3).reshape(len(a), -1) / 255.0
    
    t0 = time.time()
    logit = LogisticRegression(max_iter=3000).fit(gray32(Xtr_all[tr]), ytr_all[tr])
    rob = {k: r(float(np.mean(logit.predict(gray32(v[0])) == yte)), 4) for k, v in stress.items()}
    comparison.append({"model": "Logistic regression on pixels", "params": int(logit.coef_.size + logit.intercept_.size), "acc": rob["Original"], "robust": rob, "epochs": 0, "seconds": r(time.time() - t0, 1)})
    
    variants = {
        "Small CNN": dict(augment=False, bg=False),
        "Small CNN + augmentation": dict(augment=True, bg=False),
        "Small CNN + augmentation + new backgrounds": dict(augment=True, bg=True),
    }
  5. 5

    Quantise to int8 and re-test

    Keeps the augmented, background-trained CNN, stores its weights as int8 per output channel and re-scores every test set with the NumPy forward pass the browser mirrors.

    ml/train.py:3220 · 49 lines

    chosen = "Small CNN + augmentation + new backgrounds"
    model = trained[chosen]
    
    # export: int8 weights per output channel; evaluate with the browser's forward pass (same padding)
    def q8(w):
        scale = np.abs(w).reshape(-1, w.shape[-1]).max(axis=0) / 127
        scale = np.array([float(f"{s:.8g}") for s in scale])
        return np.round(w / scale).astype(np.int8), scale
    
    L = {ly.name: ly.get_weights() for ly in model.layers if ly.get_weights()}
    export, deq = {}, {}
    for name in ["conv1", "conv2", "conv3", "conv4", "out"]:
        w, b = L[name]
        q, scale = q8(w)
        export[name] = {"shape": list(w.shape), "scale": [float(s) for s in scale], "b": [r(x, 6) for x in b], "q": base64.b64encode(q.tobytes()).decode("ascii")}
        deq[name] = (q.astype(np.float64) * scale, np.round(b.astype(np.float64), 6))
  6. 6

    Pick demo photos

    Three random test photos per gesture, up to three mistakes, and one new-background copy per gesture, each with the shipped model's probabilities.

    ml/train.py:3271 · 17 lines

    # samples: 3 per gesture (random) + misclassified ones, plus one swapped-background copy per gesture
    def b64(a):
        return base64.b64encode(np.asarray(a, dtype=np.uint8).tobytes()).decode("ascii")
    
    rs = np.random.default_rng(SEED + 3)
    pick = [int(i) for k in range(3) for i in rs.choice(np.where(yte == k)[0], size=3, replace=False)]
    wrong = [int(i) for i in np.where(pred != yte)[0][:3]]
    samples = []
    for i in pick + [w for w in wrong if w not in pick]:
        samples.append({"id": f"T-{i:03d}", "label": int(yte[i]), "pred": int(pred[i]), "probs": [r(x, 5) for x in prob[i]], "img": b64(Xte[i]), "kind": "mistake" if pred[i] != yte[i] else "random", "variant": "original"})
    bgX = stress["New background"][0]
    bg_logits, _ = ship_eval(bgX)
    bg_prob = np.exp(bg_logits - bg_logits.max(1, keepdims=True))
    bg_prob /= bg_prob.sum(1, keepdims=True)
    for k in range(3):
        i = int(rs.choice(np.where(yte == k)[0]))
        samples.append({"id": f"B-{i:03d}", "label": int(yte[i]), "pred": int(bg_prob[i].argmax()), "probs": [r(x, 5) for x in bg_prob[i]], "img": b64(bgX[i]), "kind": "background", "variant": "new background"})
  7. 7

    Export the JSON

    Writes the int8 weights, model comparison, training history, stress results, confusion matrix and samples to src/data/models/hand-gestures.json.

    ml/train.py:3290 · 26 lines

    save(
        "hand-gestures",
        {
            "dataset": {
                "name": "Rock, Paper, Scissors (Laurence Moroney, 2019)",
                "source": "https://laurencemoroney.com/datasets.html",
                "download": "https://storage.googleapis.com/download.tensorflow.org/data/rps.zip",
                "license": "CC BY 2.0",
                "train": int(len(ytr_all)),
                "test": int(len(yte)),
                "size": 300,
            },
            "classes": CLASSES,
            "input": S,
            "split": {"train": int(len(tr)), "val": int(len(va)), "test": int(len(yte))},
            "comparison": comparison,

In your browser · TypeScript

  1. 8

    Start MediaPipe on the GPU, fall back to the CPU

    Tries the GPU delegate with a timeout and quietly retries on the CPU, so the demo starts on any device.

    src/lib/mediapipe.ts:17 · 9 lines

    export async function withGpuFallback<T>(make: (delegate: "GPU" | "CPU") => Promise<T>, timeoutMs = 8000): Promise<T> {
      // some browsers stall on GPU setup (blocked WebGL, background tabs): give it a few seconds, then use the CPU
      const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("GPU setup timed out")), timeoutMs));
      try {
        return await Promise.race([make("GPU"), timeout]);
      } catch {
        return await make("CPU");
      }
    }
  2. 9

    Smooth the gesture over the last frames

    A majority vote over recent frames, so one noisy frame can't flip rock into paper.

    src/components/demos/camera-gesture-demo.tsx:77 · 9 lines

    /** Most frequent gesture over the last frames, so a single noisy frame doesn't flip the answer. */
    function stable(history: string[]) {
      const counts = new Map<string, number>();
      for (const g of history) counts.set(g, (counts.get(g) ?? 0) + 1);
      let best = "";
      let n = 0;
      for (const [g, c] of counts) if (c > n) [best, n] = [g, c];
      return n >= Math.ceil(history.length * 0.6) ? best : "";
    }
  3. 10

    My own CNN, with a heatmap of where it looks

    Same-padding convolutions, global average pooling and a class activation map for the experiment below the game.

    src/lib/cnn.ts:137 · 36 lines

    export function createGestureNet(m: GestureCnn) {
      const W = { c1: dequant(m.conv1), c2: dequant(m.conv2), c3: dequant(m.conv3), c4: dequant(m.conv4), out: dequant(m.out) };
      const ch = [m.conv1.shape[3], m.conv2.shape[3], m.conv3.shape[3], m.conv4.shape[3]];
      const K = m.out.shape[1];
    
      /** `pixels`: size×size×3 values in 0–1 (RGB, row by row). Returns probabilities, feature maps and class activation maps. */
      return function run(pixels: ArrayLike<number>, size: number) {
        const input: Tensor = { data: Float64Array.from(pixels), h: size, w: size, c: 3 };
        const a1 = convSame(input, W.c1, m.conv1.b, ch[0]);
        const a2 = convSame(pool(a1), W.c2, m.conv2.b, ch[1]);
        const a3 = convSame(pool(a2), W.c3, m.conv3.b, ch[2]);
        const a4 = convSame(pool(a3), W.c4, m.conv4.b, ch[3]);
        const n = a4.h * a4.w;
        const gap = new Float64Array(ch[3]);
        for (let c = 0; c < ch[3]; c++) {
          let s = 0;

Notes

The decisions behind it.

Approach

  1. 1

    Ran MediaPipe's gesture recognizer (hand detector, 21-point landmark model, gesture classifier) in the browser, self-hosted, GPU first with a CPU fallback.

  2. 2

    Smoothed the per-frame gesture with a majority vote over the last 10 frames and mapped fist, palm and victory to a rock-paper-scissors game.

  3. 3

    For comparison, trained my own CNN on 2,520 rendered hands with TensorFlow: plain, with augmentation, and with background swaps, plus MobileNetV2 transfer learning.

  4. 4

    Stress-tested every CNN on rotated, darker and new-background copies of the test set; the CNN runs in the browser too, with heatmaps of where it looks.

CNN experiment: original test set · unseen backgrounds

Logistic regression on pixels63.2% · 30.6%
Small CNN95.2% · 47.0%
Small CNN + augmentation96.5% · 48.7%
Small CNN + augmentation + new backgrounds96.0% · 79.3%shipped
MobileNetV2 transfer (ImageNet) + new backgrounds98.4% · 91.4%

Honest note

The live recognizer is MediaPipe's pretrained model from Google, used as it is: I built the camera pipeline, the smoothing, the game and the comparison, not the gesture model itself. It needs a decent camera and light, and struggles with very fast motion, gloves or a hand cut off by the frame. The CNN experiment uses rendered hands only, so even its stress test is synthetic; MobileNetV2 is sturdier there (91.4% on new backgrounds) but has 37× the parameters.

Reproduce: ml/train.py → rps()

dataset source

Next project

Live Object Detection