←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

A hand-played instrument: a theremin where the right hand sets the note and the left the volume, and finger chords where fingers pick the chord and thumb taps play it.

2

Modes

97.1%

Finger count right

905

Real hands tested

none

Extra download

Overview

The theremin, invented in 1920, is played without touching it: one hand moves near an antenna to change the pitch, the other near a loop to change the volume. A webcam and a hand model can do the same job as the antennas.

This page tracks both hands and turns them into music in two ways. In Theremin mode the right hand's height picks a note on a scale and the left hand sets the volume, mutes with a fist and adds vibrato with a pinch. In Finger chords mode the left hand shows one to five fingers to pick a chord and the right thumb taps each finger to play its notes. The sound is synthesized live with the Web Audio API.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Camera

    your webcam

    Frames stay on the device

  2. 02

    Hand model

    2 × 21 points

    Google's hand landmarks, two hands

  3. 03

    Assign

    right plays

    By side of the mirrored screen

  4. 04

    Read

    height · fingers · taps

    Divided by palm length, with hysteresis

  5. 05

    Music

    scale · chord

    Snapped to a scale, five chords in a key

  6. 06

    Sound

    Web Audio

    Oscillators, filter, reverb, limiter

Inside the model

From two hands to music.

Google's networks find the hands; the music is rules I wrote on top of 21 points per hand. Scroll through the pipeline.

Hand
CAMERA FRAMEPALM DETECTOR: A HAND IS HERE

The signals

Two hands, 21 points each, turned into music.

One hand plays and the other shapes the sound, like a theremin. No samples and no music library: the notes come from oscillators the page builds in your browser.

MediaPipe hand landmarks

Google's pretrained hand model (Apache-2.0), inside the gesture recognizer bundle this site already uses for Hand Gestures and Virtual Mouse, now tracking two hands at once. Reading the hands, the music logic and the synthesizer are written for this page.

2

modes

4

scales

97.1%

finger count right

8.4 MB, shared

model download

Model page

Column dictionary

● used by the model

ColumnTypeWhat it means
middle knucklepoint 9Hand height: the note in Theremin mode (right hand) and the volume (left hand)
fingertips + middle jointspoints 6–20Which fingers are raised: the chord in Finger chords mode, a fist to mute
thumb tippoint 4Touching a fingertip plays that finger's note; touching the index adds vibrato
wrist → middle knucklepoints 0 → 9Palm length: every distance is divided by it, so it works near or far from the camera
other pointsthe rest of 21Drawn as the hand skeleton

01

Camera frame

Processed in the browser, never uploaded

02

Two hands

21 points each, from Google's hand landmark model

03

Which hand is which

By where it is on the mirrored screen: right plays, left controls

04

Gestures → music

Height to a note on a scale, raised fingers to a chord, thumb taps to notes

05

Synthesizer

Oscillators, filter, reverb and limiter built with the Web Audio API

Try it

Play music with your hands in the air.

Turn the sound on and start the camera. In Theremin mode, raise and lower your right hand for the melody and use the left for volume. In Finger chords mode, show 1 to 5 fingers on the left hand and tap your right thumb to each finger. No camera? Play with the mouse or the keyboard.

Live demo

runs in your browser

Turn your sound on. Video stays in your browser.

Playing

–

silent

Volume0

vibrato off

Mode

Scale

Pitch

Key

Sound

  • Right hand (right half of the picture): higher hand, higher note.
  • Left hand: raise the palm for louder, make a fist to mute, touch thumb to index for vibrato.
  • Only one hand? Keep it on the right: it plays at a fixed volume.
Last 6 seconds of notes · the live waveformblack: your melody

How the music works

A little music theory makes waving sound good.

An instrument you don't touch

The theremin (1920) is played in the air: one hand near an antenna sets the pitch, the other sets the volume. Here the camera is the antenna.

No wrong notes

Height is snapped to a scale. The pentatonic has no clashing intervals, so any melody sounds fine; Glide mode slides between notes like the real instrument.

Notes that don't flicker

A hand resting on the line between two notes would flip between them every frame. The current note is kept until the hand is clearly past the halfway line.

Four chords, five fingers

C, F, G, Am and Dm are the I, IV, V, vi and ii chords: enough for a huge number of pop songs. A chord must be held for four frames before it changes, so it doesn't flicker as you lift fingers.

Scale (click the keys to hear it)

5 notes per octave · coloured keys are the notes your hand can reach

1 finger · I

C

C · E · G · C

2 fingers · IV

F

C · F · A · C

3 fingers · V

G

B · D · G · B

4 fingers · vi

Am

C · E · A · C

5 fingers · ii

Dm

D · F · A · D

Tested on real hands

Finger counting is right on 97.1% of 905 real hands.

The chord depends on counting raised fingers, so I checked that rule on real hand landmarks: ASL fingerspelling letters whose hand shape shows a known number of fingers, from the same ASLNow! dataset as the Sign Language Speller.

Fingers counted correctly, by hand shape

The weakest shapes are D (pinky drifts up) and W (thumb resting on the pinky reads as raised).

0%50%100%100%A·0S·0100%E·086%D·1100%I·1100%U·2100%V·2L·2Y·2W·3F·3100%B·4
Each hand shape

Fists mute on 100% of fists; a thumb on the index is found on 97% of F hands.

LetterShapeFingersRightHands
AFist, thumb at the side0100%65
SFist, thumb across099%83
EFingers curled, thumb tucked0100%57
DIndex up186%72
IPinky up1100%68
UIndex + middle together2100%106
VIndex + middle apart2100%83
LThumb + index298%84
YThumb + pinky294%82
WThree fingers392%75
FMiddle, ring, pinky up; thumb on index397%61
BFour fingers, thumb folded4100%69
What this project is, and isn't

Google's hand model; my hand reading, music rules and synthesizer, all on your device.

Here
Hand modelGoogle's MediaPipe hand landmarks (Apache-2.0), pretrained and used as published
Built hereWhich hand is which, raised-finger counting, thumb taps, notes on a scale with stable switching, chord selection, the Web Audio synthesizer, visualizer and mouse/keyboard fallback
Test905 hands from ASLNow! fingerspelling (MIT); neighbouring video frames look alike, so the numbers are a little optimistic. ASL has no five-finger letter, so "5" isn't tested
ThresholdsThumb raised when its tip is over 0.55 palm lengths from the index knuckle; a tap below 0.26, released above 0.4; set from these hands
LatencyCamera, hand model and audio each add a little delay, so fast rhythms are hard; slow melodies and chords work best
LimitsNeeds both hands lit and in view; a hand turned sideways can hide fingers. Hands are assigned by screen side, so crossing them swaps their jobs
PrivacyVideo is processed in the browser and never uploaded or recorded; nothing is saved

Build notes

How it's built, step by step.

The real code behind this demo, taken from the repository when the site is built. Copy any piece.

In your browser · TypeScript

  1. 1

    Which hand is which

    By position on the mirrored screen, not the model's left/right label: the hand further right plays, the other controls; one hand alone takes the job of its half.

    src/lib/air-synth.ts:73 · 13 lines

    /**
     * Which hand is which, by where it is on the mirrored screen rather than by the model's left/right label:
     * the hand further right plays (pitch or notes), the other controls (volume or chord). One hand alone counts
     * as the playing hand on the right half of the screen and the control hand on the left half.
     */
    export function splitHands(hands: P[][]): { play?: P[]; control?: P[] } {
      if (hands.length >= 2) {
        const [a, b] = [...hands].sort((h1, h2) => screenX(h1) - screenX(h2));
        return { control: a, play: b };
      }
      if (hands.length === 1) return screenX(hands[0]) >= 0.5 ? { play: hands[0] } : { control: hands[0] };
      return {};
    }
  2. 2

    Hand height to a place on the scale

    The middle 80% of the frame is spread over the notes, higher hand = higher note.

    src/lib/air-synth.ts:30 · 5 lines

    /** Hand height → position along the notes (0 = lowest, n−1 = highest). `y` is 0 at the top of the frame. */
    export function heightToPos(y: number, n: number, top = BAND.top, bottom = BAND.bottom) {
      const u = Math.max(0, Math.min(1, (bottom - y) / (bottom - top)));
      return u * (n - 1);
    }
  3. 3

    Notes that don't flicker

    Snap to the nearest note, but leave the current one only when the hand is clearly past the halfway line.

    src/lib/air-synth.ts:36 · 8 lines

    /**
     * Snap a position to a note, but only leave the current note once the hand is clearly past the halfway line:
     * without this margin a hand resting on a boundary flickers between two notes.
     */
    export function stickyIndex(prev: number, pos: number, margin = 0.18) {
      if (prev < 0 || Math.abs(pos - prev) > 0.5 + margin) return Math.round(pos);
      return prev;
    }
  4. 4

    Count raised fingers

    A finger is up when its tip is farther from the wrist than its middle joint; the thumb when its tip is away from the index knuckle. Checked on real hands.

    src/lib/air-synth.ts:87 · 8 lines

    /**
     * Which fingers are raised: [thumb, index, middle, ring, pinky]. A finger is up when its tip is farther from the
     * wrist than its middle joint (works at any hand angle); the thumb when its tip is away from the index knuckle.
     */
    export function extended(lm: P[]) {
      const thumb = dist(lm[4], lm[5]) / palm(lm) > 0.55; // tuned on real hands: see scripts/check-air-synth.mjs
      return [thumb, ...TIPS.map((t, i) => dist(lm[t], lm[0]) > dist(lm[PIPS[i]], lm[0]) * 1.12)];
    }
  5. 5

    Thumb taps with hysteresis

    Gap ÷ palm length per finger; a finger stays down until it clearly opens, and a new tap must be the closest finger.

    src/lib/air-synth.ts:109 · 10 lines

    /**
     * Which finger is touching the thumb (0–3), or −1. A finger already touching stays on until it clearly opens
     * (two thresholds); a new touch must be the closest finger, because the thumb can only meet one fingertip at a time.
     */
    export function tappedFinger(prev: number, ratios: number[], on = TAP.on, off = TAP.off) {
      if (prev >= 0 && ratios[prev] < off) return prev;
      let best = -1;
      for (let i = 0; i < ratios.length; i++) if (ratios[i] < on && (best < 0 || ratios[i] < ratios[best])) best = i;
      return best;
    }
  6. 6

    Chords that wait for a steady hand

    A finger count must be seen four frames in a row before the chord changes, so lifting fingers one by one doesn't flash through chords.

    src/lib/air-synth.ts:120 · 10 lines

    /** A value must be seen this many frames in a row before it is accepted (stops chords flickering mid-gesture). */
    export function settle(state: { value: number; cand: number; n: number }, v: number, frames = 4) {
      if (v === state.cand) state.n++;
      else {
        state.cand = v;
        state.n = 1;
      }
      if (state.n >= frames) state.value = v;
      return state.value;
    }
  7. 7

    The synthesizer, in Web Audio

    A gliding lead with vibrato, four plucked voices and a bass, a low-pass tone filter, a generated reverb and a limiter, all from built-in browser nodes.

    src/lib/air-synth.ts:135 · 114 lines

    /**
     * A small synthesizer built from Web Audio nodes: one gliding lead voice for the theremin (with vibrato),
     * four plucked voices for chord notes, a bass voice, a low-pass "tone" filter, a generated reverb and a limiter.
     */
    export function createSynth() {
      const ctx = new AudioContext();
      const now = () => ctx.currentTime;
    
      const filter = ctx.createBiquadFilter(); // everything passes through one low-pass: the Tone control
      filter.type = "lowpass";
      filter.frequency.value = 2600;
      filter.Q.value = 0.8;
      const master = ctx.createGain();
      master.gain.value = 0.9;
      const limiter = ctx.createDynamicsCompressor(); // keeps four notes plus bass from clipping
      limiter.threshold.value = -14;
Data download scriptscripts/check-air-synth.mjs

scripts/check-air-synth.mjs:1 · 62 lines

/**
 * Checks Air Synth's finger reading on real hands: the ASLNow! landmark files (ml/data/raw/aslnow, see ml/train.py)
 * include hand shapes with a known number of raised fingers, a fist, and a thumb touching the index.
 * Runs the same TypeScript the page uses and writes src/data/models/air-synth.json.
 *
 *   node --experimental-strip-types scripts/check-air-synth.mjs
 */
import { readdirSync, readFileSync, writeFileSync } from "node:fs";
import { fingersUp, isFist, tapRatios, tappedFinger } from "../src/lib/air-synth.ts";

const ROOT = "ml/data/raw/aslnow";
const load = (letter) =>
  readdirSync(`${ROOT}/${letter}`)
    .filter((f) => f.endsWith(".json"))
    .sort()
    .map((f) => JSON.parse(readFileSync(`${ROOT}/${letter}/${f}`, "utf8")))

Notes

The decisions behind it.

Approach

  1. 1

    Reused the self-hosted gesture recognizer bundle for its hand landmarks, now tracking two hands, and assigned their jobs by which side of the mirrored screen they are on.

  2. 2

    Mapped the playing hand's height to a position along a scale over two octaves, snapped with a margin so a resting hand doesn't flicker between notes, or glided continuously.

  3. 3

    Counted raised fingers by comparing each fingertip's distance from the wrist with its middle joint's, and found thumb-to-finger taps from distances divided by palm length with two thresholds.

  4. 4

    Checked the counting rule on 905 real hands from ASL letters with a known number of raised fingers: 97.1% right; the thumb threshold was set from the same hands.

  5. 5

    Built the synthesizer from Web Audio nodes: a gliding lead with vibrato, four plucked chord voices and a bass, a low-pass tone control, a generated reverb and a limiter, plus a note trail and oscilloscope, and mouse and keyboard play without a camera.

Honest note

A toy instrument, not a replacement for one: the camera, the model and the audio each add a little delay, so fast rhythms are hard. The finger test uses ASL letter shapes, not people counting, and neighbouring video frames in that dataset look alike, so the score is a little optimistic; five fingers isn't covered. Hands are assigned by screen side, so crossing them swaps their jobs. The hand model is Google's; everything else is mine.

Reproduce: src/lib/air-synth.ts · src/components/demos/air-synth-demo.tsx · scripts/check-air-synth.mjs

dataset source

Next project

Handwritten Digits