Computer vision
A hand-played instrument: a theremin where the right hand sets the note and the left the volume, and finger chords where fingers pick the chord and thumb taps play it.
2
Modes
97.1%
Finger count right
905
Real hands tested
none
Extra download
Overview
The theremin, invented in 1920, is played without touching it: one hand moves near an antenna to change the pitch, the other near a loop to change the volume. A webcam and a hand model can do the same job as the antennas.
This page tracks both hands and turns them into music in two ways. In Theremin mode the right hand's height picks a note on a scale and the left hand sets the volume, mutes with a fist and adds vibrato with a pinch. In Finger chords mode the left hand shows one to five fingers to pick a chord and the right thumb taps each finger to play its notes. The sound is synthesized live with the Web Audio API.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Camera
your webcam
Frames stay on the device
- 02
Hand model
2 × 21 points
Google's hand landmarks, two hands
- 03
Assign
right plays
By side of the mirrored screen
- 04
Read
height · fingers · taps
Divided by palm length, with hysteresis
- 05
Music
scale · chord
Snapped to a scale, five chords in a key
- 06
Sound
Web Audio
Oscillators, filter, reverb, limiter
Inside the model
From two hands to music.
Google's networks find the hands; the music is rules I wrote on top of 21 points per hand. Scroll through the pipeline.
The signals
Two hands, 21 points each, turned into music.
One hand plays and the other shapes the sound, like a theremin. No samples and no music library: the notes come from oscillators the page builds in your browser.
MediaPipe hand landmarks
Google's pretrained hand model (Apache-2.0), inside the gesture recognizer bundle this site already uses for Hand Gestures and Virtual Mouse, now tracking two hands at once. Reading the hands, the music logic and the synthesizer are written for this page.
2
modes
4
scales
97.1%
finger count right
8.4 MB, shared
model download
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| middle knuckle | point 9 | Hand height: the note in Theremin mode (right hand) and the volume (left hand) |
| fingertips + middle joints | points 6–20 | Which fingers are raised: the chord in Finger chords mode, a fist to mute |
| thumb tip | point 4 | Touching a fingertip plays that finger's note; touching the index adds vibrato |
| wrist → middle knuckle | points 0 → 9 | Palm length: every distance is divided by it, so it works near or far from the camera |
| other points | the rest of 21 | Drawn as the hand skeleton |
01
Camera frame
Processed in the browser, never uploaded
02
Two hands
21 points each, from Google's hand landmark model
03
Which hand is which
By where it is on the mirrored screen: right plays, left controls
04
Gestures → music
Height to a note on a scale, raised fingers to a chord, thumb taps to notes
05
Synthesizer
Oscillators, filter, reverb and limiter built with the Web Audio API
Try it
Play music with your hands in the air.
Turn the sound on and start the camera. In Theremin mode, raise and lower your right hand for the melody and use the left for volume. In Finger chords mode, show 1 to 5 fingers on the left hand and tap your right thumb to each finger. No camera? Play with the mouse or the keyboard.
Live demo
runs in your browser
Turn your sound on. Video stays in your browser.
Playing
–
silent
vibrato off
Mode
Scale
Pitch
Key
Sound
- Right hand (right half of the picture): higher hand, higher note.
- Left hand: raise the palm for louder, make a fist to mute, touch thumb to index for vibrato.
- Only one hand? Keep it on the right: it plays at a fixed volume.
How the music works
A little music theory makes waving sound good.
An instrument you don't touch
The theremin (1920) is played in the air: one hand near an antenna sets the pitch, the other sets the volume. Here the camera is the antenna.
No wrong notes
Height is snapped to a scale. The pentatonic has no clashing intervals, so any melody sounds fine; Glide mode slides between notes like the real instrument.
Notes that don't flicker
A hand resting on the line between two notes would flip between them every frame. The current note is kept until the hand is clearly past the halfway line.
Four chords, five fingers
C, F, G, Am and Dm are the I, IV, V, vi and ii chords: enough for a huge number of pop songs. A chord must be held for four frames before it changes, so it doesn't flicker as you lift fingers.
Scale (click the keys to hear it)
5 notes per octave · coloured keys are the notes your hand can reach
1 finger · I
C
C · E · G · C
2 fingers · IV
F
C · F · A · C
3 fingers · V
G
B · D · G · B
4 fingers · vi
Am
C · E · A · C
5 fingers · ii
Dm
D · F · A · D
Tested on real hands
Finger counting is right on 97.1% of 905 real hands.
The chord depends on counting raised fingers, so I checked that rule on real hand landmarks: ASL fingerspelling letters whose hand shape shows a known number of fingers, from the same ASLNow! dataset as the Sign Language Speller.
The weakest shapes are D (pinky drifts up) and W (thumb resting on the pinky reads as raised).
Fists mute on 100% of fists; a thumb on the index is found on 97% of F hands.
| Letter | Shape | Fingers | Right | Hands |
|---|---|---|---|---|
| A | Fist, thumb at the side | 0 | 100% | 65 |
| S | Fist, thumb across | 0 | 99% | 83 |
| E | Fingers curled, thumb tucked | 0 | 100% | 57 |
| D | Index up | 1 | 86% | 72 |
| I | Pinky up | 1 | 100% | 68 |
| U | Index + middle together | 2 | 100% | 106 |
| V | Index + middle apart | 2 | 100% | 83 |
| L | Thumb + index | 2 | 98% | 84 |
| Y | Thumb + pinky | 2 | 94% | 82 |
| W | Three fingers | 3 | 92% | 75 |
| F | Middle, ring, pinky up; thumb on index | 3 | 97% | 61 |
| B | Four fingers, thumb folded | 4 | 100% | 69 |
Google's hand model; my hand reading, music rules and synthesizer, all on your device.
| Here | |
|---|---|
| Hand model | Google's MediaPipe hand landmarks (Apache-2.0), pretrained and used as published |
| Built here | Which hand is which, raised-finger counting, thumb taps, notes on a scale with stable switching, chord selection, the Web Audio synthesizer, visualizer and mouse/keyboard fallback |
| Test | 905 hands from ASLNow! fingerspelling (MIT); neighbouring video frames look alike, so the numbers are a little optimistic. ASL has no five-finger letter, so "5" isn't tested |
| Thresholds | Thumb raised when its tip is over 0.55 palm lengths from the index knuckle; a tap below 0.26, released above 0.4; set from these hands |
| Latency | Camera, hand model and audio each add a little delay, so fast rhythms are hard; slow melodies and chords work best |
| Limits | Needs both hands lit and in view; a hand turned sideways can hide fingers. Hands are assigned by screen side, so crossing them swaps their jobs |
| Privacy | Video is processed in the browser and never uploaded or recorded; nothing is saved |
Build notes
How it's built, step by step.
The real code behind this demo, taken from the repository when the site is built. Copy any piece.
In your browser · TypeScript
- 1
Which hand is which
By position on the mirrored screen, not the model's left/right label: the hand further right plays, the other controls; one hand alone takes the job of its half.
src/lib/air-synth.ts:73 · 13 lines
/** * Which hand is which, by where it is on the mirrored screen rather than by the model's left/right label: * the hand further right plays (pitch or notes), the other controls (volume or chord). One hand alone counts * as the playing hand on the right half of the screen and the control hand on the left half. */ export function splitHands(hands: P[][]): { play?: P[]; control?: P[] } { if (hands.length >= 2) { const [a, b] = [...hands].sort((h1, h2) => screenX(h1) - screenX(h2)); return { control: a, play: b }; } if (hands.length === 1) return screenX(hands[0]) >= 0.5 ? { play: hands[0] } : { control: hands[0] }; return {}; } - 2
Hand height to a place on the scale
The middle 80% of the frame is spread over the notes, higher hand = higher note.
src/lib/air-synth.ts:30 · 5 lines
/** Hand height → position along the notes (0 = lowest, n−1 = highest). `y` is 0 at the top of the frame. */ export function heightToPos(y: number, n: number, top = BAND.top, bottom = BAND.bottom) { const u = Math.max(0, Math.min(1, (bottom - y) / (bottom - top))); return u * (n - 1); } - 3
Notes that don't flicker
Snap to the nearest note, but leave the current one only when the hand is clearly past the halfway line.
src/lib/air-synth.ts:36 · 8 lines
/** * Snap a position to a note, but only leave the current note once the hand is clearly past the halfway line: * without this margin a hand resting on a boundary flickers between two notes. */ export function stickyIndex(prev: number, pos: number, margin = 0.18) { if (prev < 0 || Math.abs(pos - prev) > 0.5 + margin) return Math.round(pos); return prev; } - 4
Count raised fingers
A finger is up when its tip is farther from the wrist than its middle joint; the thumb when its tip is away from the index knuckle. Checked on real hands.
src/lib/air-synth.ts:87 · 8 lines
/** * Which fingers are raised: [thumb, index, middle, ring, pinky]. A finger is up when its tip is farther from the * wrist than its middle joint (works at any hand angle); the thumb when its tip is away from the index knuckle. */ export function extended(lm: P[]) { const thumb = dist(lm[4], lm[5]) / palm(lm) > 0.55; // tuned on real hands: see scripts/check-air-synth.mjs return [thumb, ...TIPS.map((t, i) => dist(lm[t], lm[0]) > dist(lm[PIPS[i]], lm[0]) * 1.12)]; } - 5
Thumb taps with hysteresis
Gap ÷ palm length per finger; a finger stays down until it clearly opens, and a new tap must be the closest finger.
src/lib/air-synth.ts:109 · 10 lines
/** * Which finger is touching the thumb (0–3), or −1. A finger already touching stays on until it clearly opens * (two thresholds); a new touch must be the closest finger, because the thumb can only meet one fingertip at a time. */ export function tappedFinger(prev: number, ratios: number[], on = TAP.on, off = TAP.off) { if (prev >= 0 && ratios[prev] < off) return prev; let best = -1; for (let i = 0; i < ratios.length; i++) if (ratios[i] < on && (best < 0 || ratios[i] < ratios[best])) best = i; return best; } - 6
Chords that wait for a steady hand
A finger count must be seen four frames in a row before the chord changes, so lifting fingers one by one doesn't flash through chords.
src/lib/air-synth.ts:120 · 10 lines
/** A value must be seen this many frames in a row before it is accepted (stops chords flickering mid-gesture). */ export function settle(state: { value: number; cand: number; n: number }, v: number, frames = 4) { if (v === state.cand) state.n++; else { state.cand = v; state.n = 1; } if (state.n >= frames) state.value = v; return state.value; } - 7
The synthesizer, in Web Audio
A gliding lead with vibrato, four plucked voices and a bass, a low-pass tone filter, a generated reverb and a limiter, all from built-in browser nodes.
src/lib/air-synth.ts:135 · 114 lines
/** * A small synthesizer built from Web Audio nodes: one gliding lead voice for the theremin (with vibrato), * four plucked voices for chord notes, a bass voice, a low-pass "tone" filter, a generated reverb and a limiter. */ export function createSynth() { const ctx = new AudioContext(); const now = () => ctx.currentTime; const filter = ctx.createBiquadFilter(); // everything passes through one low-pass: the Tone control filter.type = "lowpass"; filter.frequency.value = 2600; filter.Q.value = 0.8; const master = ctx.createGain(); master.gain.value = 0.9; const limiter = ctx.createDynamicsCompressor(); // keeps four notes plus bass from clipping limiter.threshold.value = -14;
Data download scriptscripts/check-air-synth.mjs
scripts/check-air-synth.mjs:1 · 62 lines
/**
* Checks Air Synth's finger reading on real hands: the ASLNow! landmark files (ml/data/raw/aslnow, see ml/train.py)
* include hand shapes with a known number of raised fingers, a fist, and a thumb touching the index.
* Runs the same TypeScript the page uses and writes src/data/models/air-synth.json.
*
* node --experimental-strip-types scripts/check-air-synth.mjs
*/
import { readdirSync, readFileSync, writeFileSync } from "node:fs";
import { fingersUp, isFist, tapRatios, tappedFinger } from "../src/lib/air-synth.ts";
const ROOT = "ml/data/raw/aslnow";
const load = (letter) =>
readdirSync(`${ROOT}/${letter}`)
.filter((f) => f.endsWith(".json"))
.sort()
.map((f) => JSON.parse(readFileSync(`${ROOT}/${letter}/${f}`, "utf8")))Notes
The decisions behind it.
Approach
- 1
Reused the self-hosted gesture recognizer bundle for its hand landmarks, now tracking two hands, and assigned their jobs by which side of the mirrored screen they are on.
- 2
Mapped the playing hand's height to a position along a scale over two octaves, snapped with a margin so a resting hand doesn't flicker between notes, or glided continuously.
- 3
Counted raised fingers by comparing each fingertip's distance from the wrist with its middle joint's, and found thumb-to-finger taps from distances divided by palm length with two thresholds.
- 4
Checked the counting rule on 905 real hands from ASL letters with a known number of raised fingers: 97.1% right; the thumb threshold was set from the same hands.
- 5
Built the synthesizer from Web Audio nodes: a gliding lead with vibrato, four plucked chord voices and a bass, a low-pass tone control, a generated reverb and a limiter, plus a note trail and oscilloscope, and mouse and keyboard play without a camera.
Honest note
A toy instrument, not a replacement for one: the camera, the model and the audio each add a little delay, so fast rhythms are hard. The finger test uses ASL letter shapes, not people counting, and neighbouring video frames in that dataset look alike, so the score is a little optimistic; five fingers isn't covered. Hands are assigned by screen side, so crossing them swaps their jobs. The hand model is Google's; everything else is mine.
Reproduce: src/lib/air-synth.ts · src/components/demos/air-synth-demo.tsx · scripts/check-air-synth.mjs
dataset sourceNext project
Handwritten Digits