←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

ASL fingerspelling read live from your webcam: hand landmarks, a small neural network, and the words you spell.

94.9%

Letters right, grouped test

96.7%

Random split (optimistic)

26

Letters

2,122

Training samples

Overview

Fingerspelling spells names and new words letter by letter with one hand, and it is where most people start learning a sign language. A webcam and a hand tracker are enough to read it: the shape of the hand is all a still letter needs.

This project trains a neural network on 2,122 hand-landmark samples from the ASLNow! dataset. In the browser, MediaPipe tracks your hand, the network reads the letter, and holding it steady types it. Tested so that near-identical frames can't leak between training and test, it gets 94.9% of letters right.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    2,122 hands

    26 letters, 21 points each

  2. 02

    Normalise

    wrist · palm

    Position and distance stop mattering

  3. 03

    Group twins

    1,312 groups

    Near-duplicates kept on one side of the split

  4. 04

    Compare

    k-NN · forest · MLP

    5-fold cross-validation, two ways

  5. 05

    Ship

    94.9%

    Neural network, grouped test

  6. 06

    Live

    hold to type

    Majority over recent frames

Inside the model

From 21 points to a letter.

Google's model finds the hand; a small neural network I trained reads the letter. Scroll through one hand, computed live with the real weights.

Average hand
CAMERA FRAME63 IN128 NEURONS64 NEURONS26 LETTERSABCDEFGHIJKLMNOPQRSTUVWXYZTOP 5 OF 26 · SOFTMAXA99.8%S0.2%Y0.0%J0.0%T0.0%LETTERS RIGHT ON UNSEEN HANDS (5-FOLD, NEAR-DUPLICATES KEPT TOGETHER)k-nearest neighbours (k=5)90.0%Random forest89.0%Neural network (MLP)94.9%

The data

2,122 hand shapes, 26 letters.

Fingerspelling spells words letter by letter with one hand. The ASLNow! project recorded people signing each letter and stored the hand landmarks, not the video, so the data is small, private and ready for a model.

ASLNow! fingerspelling

Published under the MIT licence. Each sample is the 21 points of one signing hand, captured with MediaPipe's web hand tracker from several participants. The files carry no participant id, and J and Z (moving letters in ASL) are recorded as still poses.

2,122

samples

26

letters

63

numbers per sample

1,312

near-duplicate groups

Dataset on Hugging Face

Column dictionary

● used by the model

ColumnTypeWhat it means
21 pointsx, y, z eachWrist, then four joints per finger, from MediaPipe's hand tracker in a web browser
letter1 of 26The folder the sample sits in: the letter the participant was asked to sign

01

Camera frame

Processed in the browser, never uploaded

02

21 hand points

Google's hand tracker, the same family that recorded the training data

03

Normalise

Wrist to the origin, divided by palm length: position and distance stop mattering

04

Neural network

63 numbers in, two hidden layers, one score per letter out

05

Hold to type

The letter must win most of the last frames, then be held for a moment

Try it

Spell with your hand.

Start the camera and make a letter with one hand. Hold it steady and it is typed. Practise words shows the next letter's average hand shape from the data as a hint.

Live demo

runs in your browser

Turn on your camera

Video stays on this device. Hold one hand up, palm towards the camera, and make a letter.

–

Waiting for a hand

Hold a letter for 0.9 s to type it

You spelled

HELLO

What the data looks like

The average hand for every letter.

Each sketch is the mean of all normalised samples for that letter: wrist at the bottom, fingertips in orange. They are averages of the data, not official ASL diagrams, but the shapes are already recognisable: the fist of A, the flat hand of B, the hook of C.

A

65

B

69

C

53

D

72

E

57

F

61

G

95

H

69

I

68

J

93

K

64

L

84

M

88

N

84

O

98

P

72

Q

97

R

75

S

83

T

68

U

106

V

83

W

75

X

106

Y

82

Z

155

How good is it, honestly

94.9% of letters right when near-duplicates can't leak, 96.7% on a random split.

Neighbouring frames of one recording look almost identical. On a random split they land on both sides and inflate the score, so samples closer than a small distance were grouped and kept together. The grouped number is the one to trust; a new signer may still score lower.

Three models, two ways of testing (5-fold cross-validation)

All three find the shapes; the neural network, trained on mirrored, rotated and stretched hands, is shipped because it is small and smooth to run live.

ModelRandom splitNear-duplicates grouped
k-nearest neighbours (k=5)92.9%90.0%
Random forest93.7%89.0%
Neural network (MLP) ✓96.7%94.9%
Letters it confuses most (grouped test)

The mix-ups are letters that differ by one finger or a thumb position, which ASL learners confuse too.

S read as A11×
U read as R8×
O read as E8×
V read as U7×
R read as U5×
D read as X5×
U read as V4×
P read as Z4×
Accuracy per letter (grouped test, neural network)

I and L are easiest; S and V are hardest.

0%50%100%100%I100%L100%WBFCXGNHTKJZQYPMAEODURV86%S
What this project is, and isn't

A letter speller trained from scratch on public landmark data, not a full sign language translator.

Here
ModelA dense neural network (63 → 128 → 64 → 26) trained in TensorFlow on normalised landmarks, run in TypeScript and checked against NumPy
Hand trackingGoogle's MediaPipe hand landmarks (Apache-2.0), the same tracker family that recorded the training data
ScopeASL fingerspelling only: single letters held still. Real sign languages, including Indonesia's BISINDO, use movement, both hands, the face and grammar
Limits2,122 samples from an unknown number of people; unusual hands, angles or lighting will do worse. J and Z are learned as still poses
PrivacyVideo is processed in the browser and never uploaded or recorded

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

asl() from ml/train.py + 3 helpers (r, save, _asl_normalise)

pip install numpy pandas scikit-learn scipy tensorflow · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load the landmark files

    Every sample is a JSON list of 21 hand points (x, y, z) from MediaPipe's web hand tracker, stored in a folder named after its letter.

    ml/train.py:4221 · 23 lines

    import keras
    import tensorflow as tf
    from keras import layers
    from scipy.sparse import csr_matrix
    from scipy.sparse.csgraph import connected_components
    from sklearn.model_selection import StratifiedGroupKFold
    from sklearn.neighbors import KNeighborsClassifier
    
    print("sign-language")
    keras.utils.set_random_seed(SEED)
    tf.config.experimental.enable_op_determinism()
    root = RAW / "aslnow"
    letters = sorted(p.name for p in root.iterdir() if p.is_dir() and len(p.name) == 1)
    hands, labels = [], []
    for k, letter in enumerate(letters):
        for f in sorted((root / letter).glob("*.json")):
  2. 2

    Normalise every hand

    Move the wrist to the origin and divide by the palm's length, so position in the frame and distance to the camera stop mattering.

    ml/train.py:4246 · 2 lines

    X = _asl_normalise(H)
    Xf = X.reshape(len(X), -1)
  3. 3

    Group near-duplicate samples

    The files carry no participant id and neighbouring video frames look alike, so samples closer than a small distance are grouped and never split between train and test.

    ml/train.py:4250 · 5 lines

    D = np.sqrt(((Xf[:, None, :] - Xf[None, :, :]) ** 2).sum(-1))
    np.fill_diagonal(D, np.inf)
    eps = float(np.quantile(D.min(1), 0.5))  # half of all samples have a twin at least this close
    n_groups, groups = connected_components(csr_matrix(D < eps), directed=False)
    print(f"  near-duplicate threshold {eps:.4f}: {n_groups} groups, largest {np.bincount(groups).max()} samples")
  4. 4

    Define the models

    k-nearest neighbours and a random forest as classic baselines, and a small neural network trained with mirrored, rotated and stretched hands.

    ml/train.py:4257 · 34 lines

    def augment(A, yy, copies=6, seed=SEED):
        rng = np.random.default_rng(seed)
        out, ys = [A], [yy]
        for _ in range(copies):
            B = A.copy()
            flip = rng.random(len(B)) < 0.5  # left or right hand
            B[flip, :, 0] *= -1
            th = rng.uniform(-0.3, 0.3, len(B))  # roll the hand up to about 17 degrees
            c, s = np.cos(th)[:, None], np.sin(th)[:, None]
            x0, y0 = B[:, :, 0].copy(), B[:, :, 1].copy()
            B[:, :, 0], B[:, :, 1] = c * x0 - s * y0, s * x0 + c * y0
            B[:, :, 0] *= rng.uniform(0.75, 1.33, len(B))[:, None]  # widescreen vs 4:3 cameras squeeze x differently
            B += rng.normal(0, 0.02, B.shape)
            out.append(B)
            ys.append(yy)
        return np.concatenate(out).reshape(-1, 63).astype("float32"), np.concatenate(ys)
  5. 5

    Evaluate two ways

    Stratified 5-fold cross-validation on a random split and on the near-duplicate groups; the grouped score is the honest one.

    ml/train.py:4293 · 17 lines

    results = {}
    conf = np.zeros((len(letters), len(letters)), dtype=int)
    for split_name, splitter, g in [("random", StratifiedKFold(5, shuffle=True, random_state=SEED), None), ("grouped", StratifiedGroupKFold(5, shuffle=True, random_state=SEED), groups)]:
        accs = {"k-nearest neighbours (k=5)": [], "Random forest": [], "Neural network (MLP)": []}
        for tr, te in splitter.split(Xf, y, g):
            knn = KNeighborsClassifier(5).fit(mirror_both(X[tr]).reshape(-1, 63), np.concatenate([y[tr], y[tr]]))
            accs["k-nearest neighbours (k=5)"].append(float(np.mean(knn.predict(Xf[te]) == y[te])))
            rf = RandomForestClassifier(n_estimators=300, random_state=SEED, n_jobs=-1).fit(*augment(X[tr], y[tr], copies=3))
            accs["Random forest"].append(float(np.mean(rf.predict(Xf[te]) == y[te])))
            m = fit_mlp(X[tr], y[tr])
            p = m.predict(Xf[te].astype("float32"), verbose=0).argmax(1)
            accs["Neural network (MLP)"].append(float(np.mean(p == y[te])))
            if split_name == "grouped":
                for a, b in zip(y[te], p):
                    conf[a, b] += 1
        results[split_name] = {k: r(np.mean(v), 4) for k, v in accs.items()}
        print(f"  {split_name}: " + ", ".join(f"{k} {np.mean(v):.4f}" for k, v in accs.items()))
  6. 6

    Train the final network and mirror it in NumPy

    Fit the MLP on every sample, export its weights as plain numbers and check that the browser's forward pass gives the same answers.

    ml/train.py:4312 · 17 lines

    final = fit_mlp(X, y)
    Lw = {ly.name: ly.get_weights() for ly in final.layers if ly.get_weights()}
    rnd = lambda a: [float(f"{v:.6g}") for v in np.asarray(a).ravel()]  # noqa: E731
    export = {name: {"shape": list(Lw[name][0].shape), "w": rnd(Lw[name][0]), "b": rnd(Lw[name][1])} for name in ["d1", "d2", "out"]}
    
    def forward(F):
        a = F.astype(np.float64)
        for name, act in [("d1", True), ("d2", True), ("out", False)]:
            W = np.array(export[name]["w"]).reshape(export[name]["shape"])
            a = a @ W + np.array(export[name]["b"])
            if act:
                a = np.maximum(a, 0)
        return a
    
    gap = float(np.abs(forward(Xf[:300]) - final.predict(Xf[:300].astype("float32"), verbose=0)).max())
    print(f"  keras vs numpy forward pass: max gap {gap:.2e}")
    json.dump({"X": Xf[::5][:400].round(6).tolist(), "logits": forward(Xf[::5][:400].round(6)).tolist()}, open(RAW / "asl_parity.json", "w"))
  7. 7

    Export the JSON

    Letters, counts, both evaluations, the grouped confusion matrix, the average hand per letter for the reference chart and the network weights.

    ml/train.py:4331 · 29 lines

    per_letter = [r(conf[k, k] / max(1, conf[k].sum()), 4) for k in range(len(letters))]
    pairs = sorted([(int(conf[a, b]), letters[a], letters[b]) for a in range(len(letters)) for b in range(len(letters)) if a != b], reverse=True)[:10]
    # for the reference sketches only: put every hand the same way round (index knuckle right of the pinky's) before averaging,
    # otherwise left and right hands cancel out sideways
    canon = X.copy()
    canon[canon[:, 5, 0] < canon[:, 17, 0], :, 0] *= -1
    mean_hand = [np.round(canon[y == k].mean(0)[:, :2], 4).tolist() for k in range(len(letters))]
    knuckles = float(np.median(np.abs(canon[:, 5, 0] - canon[:, 17, 0])))  # knuckle span / palm length, about 0.8 on a real hand
    save(
        "sign-language",
        {
            "dataset": {"name": "ASLNow! fingerspelling", "source": "https://huggingface.co/datasets/sid220/asl-now-fingerspelling", "license": "MIT", "samples": int(len(X)), "commit": "9b3c96ae"},
            "letters": letters,
            "counts": [int((y == k).sum()) for k in range(len(letters))],
            "groups": int(n_groups),
            "eps": r(eps, 5),

In your browser · TypeScript

  1. 8

    Normalise the hand like in training

    Wrist to the origin and every point divided by the palm's length, exactly as _asl_normalise() in Python.

    src/lib/sign.ts:9 · 10 lines

    /**
     * 21 points → 63 numbers: wrist moved to (0, 0, 0), everything divided by the wrist-to-middle-knuckle length (x, y only).
     * `xScale` puts x in the training data's units: landmarks are divided by the frame's width, so a 4:3 webcam gives wider
     * hands than the widescreen camera the data came from; pass (videoWidth / videoHeight) / dataAspect. Training uses 1.
     */
    export function normalise(pts: Pt[], xScale = 1): number[] {
      const o = pts[0];
      const palm = Math.max(1e-6, Math.hypot((pts[9].x - o.x) * xScale, pts[9].y - o.y));
      return pts.flatMap((p) => [((p.x - o.x) * xScale) / palm, (p.y - o.y) / palm, (p.z - o.z) / palm]);
    }
  2. 9

    The neural network in TypeScript

    Three dense layers with ReLU, from the exported weights; checked against NumPy on 400 samples.

    src/lib/sign.ts:31 · 4 lines

    /** Logits for every letter, exactly as the NumPy reference in ml/train.py. */
    export function signLogits(m: SignModel, features: number[]) {
      return dense(dense(dense(features, m.d1, true), m.d2, true), m.out, false);
    }
  3. 10

    Only type a letter that holds

    The majority letter over the last frames and its share; a letter is typed once it wins most frames for almost a second.

    src/lib/sign.ts:43 · 9 lines

    /** Majority letter over recent frames, with its share; used to type a letter only once it is held steady. */
    export function steady(history: number[]): { k: number; share: number } {
      const counts = new Map<number, number>();
      for (const k of history) counts.set(k, (counts.get(k) ?? 0) + 1);
      let best = -1;
      let n = 0;
      for (const [k, c] of counts) if (c > n) [best, n] = [k, c];
      return { k: best, share: history.length ? n / history.length : 0 };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Loaded every sample's 21 hand points and normalised them: wrist at the origin, divided by the palm's length.

  2. 2

    Grouped near-identical samples (1,312 groups), because the files have no participant id and neighbouring frames of one recording look alike.

  3. 3

    Compared k-nearest neighbours, a random forest and a small neural network with stratified 5-fold cross-validation, on a random split and on the groups; trained with mirrored, rotated and stretched hands.

  4. 4

    Shipped the neural network as plain numbers, checked the TypeScript forward pass against NumPy, and built live spelling and word practice on MediaPipe hand tracking.

5-fold accuracy · random split · near-duplicates grouped

k-nearest neighbours (k=5)92.9% · 90.0%
Random forest93.7% · 89.0%
Neural network (MLP)96.7% · 94.9%shipped

Honest note

This spells ASL letters; it doesn't translate sign language, which uses movement, both hands, the face and its own grammar (Indonesia's BISINDO included). The dataset is small and its number of signers unknown, so a new signer may score below the 94.9% grouped result, and J and Z are learned as still poses although they move in ASL. Its most common mix-up is S read as A.

Reproduce: ml/train.py → asl()

dataset source

Next project

Traffic Counter