Computer vision
ASL fingerspelling read live from your webcam: hand landmarks, a small neural network, and the words you spell.
94.9%
Letters right, grouped test
96.7%
Random split (optimistic)
26
Letters
2,122
Training samples
Overview
Fingerspelling spells names and new words letter by letter with one hand, and it is where most people start learning a sign language. A webcam and a hand tracker are enough to read it: the shape of the hand is all a still letter needs.
This project trains a neural network on 2,122 hand-landmark samples from the ASLNow! dataset. In the browser, MediaPipe tracks your hand, the network reads the letter, and holding it steady types it. Tested so that near-identical frames can't leak between training and test, it gets 94.9% of letters right.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
2,122 hands
26 letters, 21 points each
- 02
Normalise
wrist · palm
Position and distance stop mattering
- 03
Group twins
1,312 groups
Near-duplicates kept on one side of the split
- 04
Compare
k-NN · forest · MLP
5-fold cross-validation, two ways
- 05
Ship
94.9%
Neural network, grouped test
- 06
Live
hold to type
Majority over recent frames
Inside the model
From 21 points to a letter.
Google's model finds the hand; a small neural network I trained reads the letter. Scroll through one hand, computed live with the real weights.
The data
2,122 hand shapes, 26 letters.
Fingerspelling spells words letter by letter with one hand. The ASLNow! project recorded people signing each letter and stored the hand landmarks, not the video, so the data is small, private and ready for a model.
ASLNow! fingerspelling
Published under the MIT licence. Each sample is the 21 points of one signing hand, captured with MediaPipe's web hand tracker from several participants. The files carry no participant id, and J and Z (moving letters in ASL) are recorded as still poses.
2,122
samples
26
letters
63
numbers per sample
1,312
near-duplicate groups
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| 21 points | x, y, z each | Wrist, then four joints per finger, from MediaPipe's hand tracker in a web browser |
| letter | 1 of 26 | The folder the sample sits in: the letter the participant was asked to sign |
01
Camera frame
Processed in the browser, never uploaded
02
21 hand points
Google's hand tracker, the same family that recorded the training data
03
Normalise
Wrist to the origin, divided by palm length: position and distance stop mattering
04
Neural network
63 numbers in, two hidden layers, one score per letter out
05
Hold to type
The letter must win most of the last frames, then be held for a moment
Try it
Spell with your hand.
Start the camera and make a letter with one hand. Hold it steady and it is typed. Practise words shows the next letter's average hand shape from the data as a hint.
Live demo
runs in your browser
Turn on your camera
Video stays on this device. Hold one hand up, palm towards the camera, and make a letter.
Waiting for a hand
Hold a letter for 0.9 s to type it
You spelled
HELLO
What the data looks like
The average hand for every letter.
Each sketch is the mean of all normalised samples for that letter: wrist at the bottom, fingertips in orange. They are averages of the data, not official ASL diagrams, but the shapes are already recognisable: the fist of A, the flat hand of B, the hook of C.
A
65
B
69
C
53
D
72
E
57
F
61
G
95
H
69
I
68
J
93
K
64
L
84
M
88
N
84
O
98
P
72
Q
97
R
75
S
83
T
68
U
106
V
83
W
75
X
106
Y
82
Z
155
How good is it, honestly
94.9% of letters right when near-duplicates can't leak, 96.7% on a random split.
Neighbouring frames of one recording look almost identical. On a random split they land on both sides and inflate the score, so samples closer than a small distance were grouped and kept together. The grouped number is the one to trust; a new signer may still score lower.
All three find the shapes; the neural network, trained on mirrored, rotated and stretched hands, is shipped because it is small and smooth to run live.
| Model | Random split | Near-duplicates grouped |
|---|---|---|
| k-nearest neighbours (k=5) | 92.9% | 90.0% |
| Random forest | 93.7% | 89.0% |
| Neural network (MLP) ✓ | 96.7% | 94.9% |
The mix-ups are letters that differ by one finger or a thumb position, which ASL learners confuse too.
I and L are easiest; S and V are hardest.
A letter speller trained from scratch on public landmark data, not a full sign language translator.
| Here | |
|---|---|
| Model | A dense neural network (63 → 128 → 64 → 26) trained in TensorFlow on normalised landmarks, run in TypeScript and checked against NumPy |
| Hand tracking | Google's MediaPipe hand landmarks (Apache-2.0), the same tracker family that recorded the training data |
| Scope | ASL fingerspelling only: single letters held still. Real sign languages, including Indonesia's BISINDO, use movement, both hands, the face and grammar |
| Limits | 2,122 samples from an unknown number of people; unusual hands, angles or lighting will do worse. J and Z are learned as still poses |
| Privacy | Video is processed in the browser and never uploaded or recorded |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
asl() from ml/train.py + 3 helpers (r, save, _asl_normalise)
pip install numpy pandas scikit-learn scipy tensorflow · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load the landmark files
Every sample is a JSON list of 21 hand points (x, y, z) from MediaPipe's web hand tracker, stored in a folder named after its letter.
ml/train.py:4221 · 23 lines
import keras import tensorflow as tf from keras import layers from scipy.sparse import csr_matrix from scipy.sparse.csgraph import connected_components from sklearn.model_selection import StratifiedGroupKFold from sklearn.neighbors import KNeighborsClassifier print("sign-language") keras.utils.set_random_seed(SEED) tf.config.experimental.enable_op_determinism() root = RAW / "aslnow" letters = sorted(p.name for p in root.iterdir() if p.is_dir() and len(p.name) == 1) hands, labels = [], [] for k, letter in enumerate(letters): for f in sorted((root / letter).glob("*.json")): - 2
Normalise every hand
Move the wrist to the origin and divide by the palm's length, so position in the frame and distance to the camera stop mattering.
ml/train.py:4246 · 2 lines
X = _asl_normalise(H) Xf = X.reshape(len(X), -1) - 3
Group near-duplicate samples
The files carry no participant id and neighbouring video frames look alike, so samples closer than a small distance are grouped and never split between train and test.
ml/train.py:4250 · 5 lines
D = np.sqrt(((Xf[:, None, :] - Xf[None, :, :]) ** 2).sum(-1)) np.fill_diagonal(D, np.inf) eps = float(np.quantile(D.min(1), 0.5)) # half of all samples have a twin at least this close n_groups, groups = connected_components(csr_matrix(D < eps), directed=False) print(f" near-duplicate threshold {eps:.4f}: {n_groups} groups, largest {np.bincount(groups).max()} samples") - 4
Define the models
k-nearest neighbours and a random forest as classic baselines, and a small neural network trained with mirrored, rotated and stretched hands.
ml/train.py:4257 · 34 lines
def augment(A, yy, copies=6, seed=SEED): rng = np.random.default_rng(seed) out, ys = [A], [yy] for _ in range(copies): B = A.copy() flip = rng.random(len(B)) < 0.5 # left or right hand B[flip, :, 0] *= -1 th = rng.uniform(-0.3, 0.3, len(B)) # roll the hand up to about 17 degrees c, s = np.cos(th)[:, None], np.sin(th)[:, None] x0, y0 = B[:, :, 0].copy(), B[:, :, 1].copy() B[:, :, 0], B[:, :, 1] = c * x0 - s * y0, s * x0 + c * y0 B[:, :, 0] *= rng.uniform(0.75, 1.33, len(B))[:, None] # widescreen vs 4:3 cameras squeeze x differently B += rng.normal(0, 0.02, B.shape) out.append(B) ys.append(yy) return np.concatenate(out).reshape(-1, 63).astype("float32"), np.concatenate(ys) - 5
Evaluate two ways
Stratified 5-fold cross-validation on a random split and on the near-duplicate groups; the grouped score is the honest one.
ml/train.py:4293 · 17 lines
results = {} conf = np.zeros((len(letters), len(letters)), dtype=int) for split_name, splitter, g in [("random", StratifiedKFold(5, shuffle=True, random_state=SEED), None), ("grouped", StratifiedGroupKFold(5, shuffle=True, random_state=SEED), groups)]: accs = {"k-nearest neighbours (k=5)": [], "Random forest": [], "Neural network (MLP)": []} for tr, te in splitter.split(Xf, y, g): knn = KNeighborsClassifier(5).fit(mirror_both(X[tr]).reshape(-1, 63), np.concatenate([y[tr], y[tr]])) accs["k-nearest neighbours (k=5)"].append(float(np.mean(knn.predict(Xf[te]) == y[te]))) rf = RandomForestClassifier(n_estimators=300, random_state=SEED, n_jobs=-1).fit(*augment(X[tr], y[tr], copies=3)) accs["Random forest"].append(float(np.mean(rf.predict(Xf[te]) == y[te]))) m = fit_mlp(X[tr], y[tr]) p = m.predict(Xf[te].astype("float32"), verbose=0).argmax(1) accs["Neural network (MLP)"].append(float(np.mean(p == y[te]))) if split_name == "grouped": for a, b in zip(y[te], p): conf[a, b] += 1 results[split_name] = {k: r(np.mean(v), 4) for k, v in accs.items()} print(f" {split_name}: " + ", ".join(f"{k} {np.mean(v):.4f}" for k, v in accs.items())) - 6
Train the final network and mirror it in NumPy
Fit the MLP on every sample, export its weights as plain numbers and check that the browser's forward pass gives the same answers.
ml/train.py:4312 · 17 lines
final = fit_mlp(X, y) Lw = {ly.name: ly.get_weights() for ly in final.layers if ly.get_weights()} rnd = lambda a: [float(f"{v:.6g}") for v in np.asarray(a).ravel()] # noqa: E731 export = {name: {"shape": list(Lw[name][0].shape), "w": rnd(Lw[name][0]), "b": rnd(Lw[name][1])} for name in ["d1", "d2", "out"]} def forward(F): a = F.astype(np.float64) for name, act in [("d1", True), ("d2", True), ("out", False)]: W = np.array(export[name]["w"]).reshape(export[name]["shape"]) a = a @ W + np.array(export[name]["b"]) if act: a = np.maximum(a, 0) return a gap = float(np.abs(forward(Xf[:300]) - final.predict(Xf[:300].astype("float32"), verbose=0)).max()) print(f" keras vs numpy forward pass: max gap {gap:.2e}") json.dump({"X": Xf[::5][:400].round(6).tolist(), "logits": forward(Xf[::5][:400].round(6)).tolist()}, open(RAW / "asl_parity.json", "w")) - 7
Export the JSON
Letters, counts, both evaluations, the grouped confusion matrix, the average hand per letter for the reference chart and the network weights.
ml/train.py:4331 · 29 lines
per_letter = [r(conf[k, k] / max(1, conf[k].sum()), 4) for k in range(len(letters))] pairs = sorted([(int(conf[a, b]), letters[a], letters[b]) for a in range(len(letters)) for b in range(len(letters)) if a != b], reverse=True)[:10] # for the reference sketches only: put every hand the same way round (index knuckle right of the pinky's) before averaging, # otherwise left and right hands cancel out sideways canon = X.copy() canon[canon[:, 5, 0] < canon[:, 17, 0], :, 0] *= -1 mean_hand = [np.round(canon[y == k].mean(0)[:, :2], 4).tolist() for k in range(len(letters))] knuckles = float(np.median(np.abs(canon[:, 5, 0] - canon[:, 17, 0]))) # knuckle span / palm length, about 0.8 on a real hand save( "sign-language", { "dataset": {"name": "ASLNow! fingerspelling", "source": "https://huggingface.co/datasets/sid220/asl-now-fingerspelling", "license": "MIT", "samples": int(len(X)), "commit": "9b3c96ae"}, "letters": letters, "counts": [int((y == k).sum()) for k in range(len(letters))], "groups": int(n_groups), "eps": r(eps, 5),
In your browser · TypeScript
- 8
Normalise the hand like in training
Wrist to the origin and every point divided by the palm's length, exactly as _asl_normalise() in Python.
src/lib/sign.ts:9 · 10 lines
/** * 21 points → 63 numbers: wrist moved to (0, 0, 0), everything divided by the wrist-to-middle-knuckle length (x, y only). * `xScale` puts x in the training data's units: landmarks are divided by the frame's width, so a 4:3 webcam gives wider * hands than the widescreen camera the data came from; pass (videoWidth / videoHeight) / dataAspect. Training uses 1. */ export function normalise(pts: Pt[], xScale = 1): number[] { const o = pts[0]; const palm = Math.max(1e-6, Math.hypot((pts[9].x - o.x) * xScale, pts[9].y - o.y)); return pts.flatMap((p) => [((p.x - o.x) * xScale) / palm, (p.y - o.y) / palm, (p.z - o.z) / palm]); } - 9
The neural network in TypeScript
Three dense layers with ReLU, from the exported weights; checked against NumPy on 400 samples.
src/lib/sign.ts:31 · 4 lines
/** Logits for every letter, exactly as the NumPy reference in ml/train.py. */ export function signLogits(m: SignModel, features: number[]) { return dense(dense(dense(features, m.d1, true), m.d2, true), m.out, false); } - 10
Only type a letter that holds
The majority letter over the last frames and its share; a letter is typed once it wins most frames for almost a second.
src/lib/sign.ts:43 · 9 lines
/** Majority letter over recent frames, with its share; used to type a letter only once it is held steady. */ export function steady(history: number[]): { k: number; share: number } { const counts = new Map<number, number>(); for (const k of history) counts.set(k, (counts.get(k) ?? 0) + 1); let best = -1; let n = 0; for (const [k, c] of counts) if (c > n) [best, n] = [k, c]; return { k: best, share: history.length ? n / history.length : 0 }; }
Notes
The decisions behind it.
Approach
- 1
Loaded every sample's 21 hand points and normalised them: wrist at the origin, divided by the palm's length.
- 2
Grouped near-identical samples (1,312 groups), because the files have no participant id and neighbouring frames of one recording look alike.
- 3
Compared k-nearest neighbours, a random forest and a small neural network with stratified 5-fold cross-validation, on a random split and on the groups; trained with mirrored, rotated and stretched hands.
- 4
Shipped the neural network as plain numbers, checked the TypeScript forward pass against NumPy, and built live spelling and word practice on MediaPipe hand tracking.
5-fold accuracy · random split · near-duplicates grouped
Honest note
This spells ASL letters; it doesn't translate sign language, which uses movement, both hands, the face and its own grammar (Indonesia's BISINDO included). The dataset is small and its number of signers unknown, so a new signer may score below the 94.9% grouped result, and J and Z are learned as still poses although they move in ASL. Its most common mix-up is S read as A.
Reproduce: ml/train.py → asl()
dataset sourceNext project
Traffic Counter