Computer vision
A floating cursor you steer with your hand: point to move, pinch to click and drag, all from the webcam.
21
Hand points tracked
4
Points that drive it
none
Extra download
move · click · drag
Actions
Overview
Touch-free control matters in kitchens, operating theatres, kiosks and for people who can't use a mouse. Hand tracking makes it possible with an ordinary webcam: the joints of the hand are found many times a second, and a couple of fingertips are enough to act as a pointer.
This project turns MediaPipe's 21 hand points into a cursor. The pointer sits between the thumb and index fingertip, is smoothed with a One Euro filter, and a pinch measured relative to the palm becomes a click or a drag. The events go to ordinary widgets in a playground: buttons, a slider, a draggable sticker, a drawing pad and a bubble-popping challenge.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Camera
your webcam
Frames stay on the device
- 02
Hand points
21 landmarks
Google's hand model, GPU first
- 03
Cursor
thumb–index midpoint
Mirrored, from the middle of the view
- 04
Smooth
One Euro filter
Still = steady, fast = responsive
- 05
Pinch
< 0.28 · > 0.42
Gap ÷ palm length, with hysteresis
- 06
Events
down · move · up · click
Real pointer events on page widgets
Inside the model
From a hand to a click.
Two pretrained networks from Google find the hand and its 21 points; everything after that is plain geometry I wrote. Scroll through the pipeline.
The signal
Twenty-one points on your hand, turned into a mouse.
Hand tracking gives the position of every joint, many times a second. Two fingertips and the palm's length are enough for a cursor you can move, click and drag with.
MediaPipe hand landmarks
Google's pretrained hand landmark model (Apache-2.0), packed inside the gesture recognizer bundle this site already uses for Hand Gestures, so the cursor costs no extra download. It returns 21 points per hand; everything from point to cursor and click is written for this page.
21
hand points
4
points used
< 0.28
pinch closes
8.4 MB
model (shared)
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| thumb tip | point 4 | One end of the pinch; the cursor sits halfway between it and the index tip |
| index tip | point 8 | The other end of the pinch |
| wrist → middle knuckle | points 0 → 9 | Palm length: divides the pinch gap so it works near or far from the camera |
| other points | 18 more | Drawn as the hand skeleton in the preview |
01
Camera frame
Processed in the browser, never uploaded
02
21 hand points
Google's hand landmark model, inside the gesture recognizer bundle
03
Cursor point
Midpoint of thumb and index tips, mirrored, mapped from the middle of the view
04
One Euro filter
Smooth when the hand is still, quick when it moves
05
Pinch → events
Gap ÷ palm length with hysteresis becomes pointerdown, move, up and click
Try it
Point, pinch, drag. No mouse needed.
Start the camera and hold up one hand. Change the colour, slide the size, drag the sticker, pop eight bubbles against the clock or draw on the pad, all with a pinch.
Live demo
runs in your browser
How to use it. Hold one hand up, palm towards the camera, about an arm's length away. The cursor sits between your thumb and index fingertip.
- Move: move your hand; the middle of the camera view covers the whole playground.
- Click: pinch thumb and index together and let go quickly.
- Drag: pinch, move, then release; it works on the slider, the sticker and the drawing pad.
Browsers don't let a web page move your computer's real mouse, so the hand cursor works inside the playground below. Your mouse works there too.
Drag the sticker · pop the bubbles
The details that make it usable
Raw hand points are jittery. Four small decisions fix that.
01
Cursor between the fingertips
When you pinch, both tips move towards each other, so their midpoint stays put: the click lands where you aimed.
02
Pinch relative to the palm
Dividing the fingertip gap by the palm's length gives the same number whether your hand is close to the camera or far away.
03
Two thresholds, not one
Pinch closes below 0.28 and opens only above 0.42. With one threshold, a hand resting near it would click on and off.
04
Only the middle of the view
The box from 20% to 80% across and 15% to 75% down covers the whole playground, so you never reach for the frame's edge, where tracking is weakest.
Still, quick move, still. Raise beta and the filter follows fast moves sooner; lower it and it smooths more but lags. The demo uses beta 0.02.
Honest notes
A mouse inside the page, not your computer's.
Browsers don't let websites move the system cursor or click outside the page, for good security reasons. A desktop version would send the same events to the operating system, for example with Python and PyAutoGUI.
Google's hand model, my cursor, filter, pinch logic and playground, on your device.
| Here | |
|---|---|
| Hand model | Google's MediaPipe hand landmarks (Apache-2.0), pretrained and used as published |
| Built here | Cursor mapping, One Euro smoothing, pinch with hysteresis, synthetic pointer and click events, and the playground |
| Accuracy | Not benchmarked; thresholds were tuned by hand. Speed depends on your device and is shown live |
| Limits | One hand, good light, palm facing the camera. Holding an arm up gets tiring after a minute or two, which is why hand-tracked mice suit short tasks |
| Privacy | Video is processed in the browser and never uploaded or recorded |
Build notes
How it's built, step by step.
The real code behind this demo, taken from the repository when the site is built. Copy any piece.
In your browser · TypeScript
- 1
Track the hand
Reuses the gesture recognizer bundle for its 21 hand landmarks, GPU first, CPU if the GPU stalls.
src/lib/mediapipe.ts:50 · 11 lines
/** Hand tracking for the cursor: the gesture recognizer already returns the 21 hand landmarks, so no extra model file. */ export async function loadHandTracker(numHands = 1): Promise<GestureRecognizer> { const [{ GestureRecognizer: GR }, vision] = await Promise.all([import("@mediapipe/tasks-vision"), visionFileset()]); return withGpuFallback((delegate) => GR.createFromOptions(vision, { baseOptions: { modelAssetPath: "/mediapipe/gesture_recognizer.task", delegate }, runningMode: "VIDEO", numHands, }), ); } - 2
Map the hand to the playground
The comfortable middle of the camera view is stretched over the whole playground, so you never reach for the edges.
src/lib/virtual-mouse.ts:22 · 5 lines
export function toArea(p: P, w: number, h: number, box = BOX): P { const u = Math.max(0, Math.min(1, (p.x - box.x0) / (box.x1 - box.x0))); const v = Math.max(0, Math.min(1, (p.y - box.y0) / (box.y1 - box.y0))); return { x: u * w, y: v * h }; } - 3
Pinch with hysteresis
Closes below one threshold and opens only above a higher one, so a hand near the boundary doesn't click on and off.
src/lib/virtual-mouse.ts:30 · 3 lines
export function nextPinch(pinched: boolean, ratio: number) { return pinched ? ratio < PINCH.up : ratio < PINCH.down; } - 4
Send real pointer events
Synthetic PointerEvent and MouseEvent objects, dispatched to the element under the cursor so ordinary widgets respond.
src/components/demos/virtual-mouse-demo.tsx:46 · 5 lines
/** Synthetic pointer / mouse events, so ordinary page widgets react to the hand cursor. */ function fire(el: Element, type: string, x: number, y: number, down: boolean) { const init = { bubbles: true, cancelable: true, clientX: x, clientY: y, pointerId: 77, pointerType: "mouse", isPrimary: true, button: 0, buttons: down ? 1 : 0 }; el.dispatchEvent(type === "click" ? new MouseEvent("click", init) : new PointerEvent(type, init)); }
Notes
The decisions behind it.
Approach
- 1
Reused the self-hosted MediaPipe gesture recognizer for its 21 hand landmarks, GPU first with a CPU fallback.
- 2
Put the cursor at the midpoint of the thumb and index tips (stable while pinching), mirrored it and mapped the comfortable middle of the camera view to the whole playground.
- 3
Smoothed the cursor with a One Euro filter and detected the pinch as fingertip gap over palm length, closing and opening at different thresholds.
- 4
Sent synthetic pointer and click events to whatever sits under the cursor, so ordinary widgets respond, and built a playground to try it.
Honest note
Browsers don't let a web page move the computer's real cursor or click outside the page, so this mouse works inside the playground; a desktop version would send the same events to the operating system. The hand model is Google's; the thresholds were tuned by hand, not benchmarked. It needs one hand in good light, and holding an arm up is tiring after a minute or two.
Reproduce: src/lib/virtual-mouse.ts · src/components/demos/virtual-mouse-demo.tsx
dataset sourceNext project
Sign Language Speller