←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

A floating cursor you steer with your hand: point to move, pinch to click and drag, all from the webcam.

21

Hand points tracked

4

Points that drive it

none

Extra download

move · click · drag

Actions

Overview

Touch-free control matters in kitchens, operating theatres, kiosks and for people who can't use a mouse. Hand tracking makes it possible with an ordinary webcam: the joints of the hand are found many times a second, and a couple of fingertips are enough to act as a pointer.

This project turns MediaPipe's 21 hand points into a cursor. The pointer sits between the thumb and index fingertip, is smoothed with a One Euro filter, and a pinch measured relative to the palm becomes a click or a drag. The events go to ordinary widgets in a playground: buttons, a slider, a draggable sticker, a drawing pad and a bubble-popping challenge.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Camera

    your webcam

    Frames stay on the device

  2. 02

    Hand points

    21 landmarks

    Google's hand model, GPU first

  3. 03

    Cursor

    thumb–index midpoint

    Mirrored, from the middle of the view

  4. 04

    Smooth

    One Euro filter

    Still = steady, fast = responsive

  5. 05

    Pinch

    < 0.28 · > 0.42

    Gap ÷ palm length, with hysteresis

  6. 06

    Events

    down · move · up · click

    Real pointer events on page widgets

Inside the model

From a hand to a click.

Two pretrained networks from Google find the hand and its 21 points; everything after that is plain geometry I wrote. Scroll through the pipeline.

Hand
CAMERA FRAMEPALM DETECTOR: A HAND IS HERE

The signal

Twenty-one points on your hand, turned into a mouse.

Hand tracking gives the position of every joint, many times a second. Two fingertips and the palm's length are enough for a cursor you can move, click and drag with.

MediaPipe hand landmarks

Google's pretrained hand landmark model (Apache-2.0), packed inside the gesture recognizer bundle this site already uses for Hand Gestures, so the cursor costs no extra download. It returns 21 points per hand; everything from point to cursor and click is written for this page.

21

hand points

4

points used

< 0.28

pinch closes

8.4 MB

model (shared)

Model page

Column dictionary

● used by the model

ColumnTypeWhat it means
thumb tippoint 4One end of the pinch; the cursor sits halfway between it and the index tip
index tippoint 8The other end of the pinch
wrist → middle knucklepoints 0 → 9Palm length: divides the pinch gap so it works near or far from the camera
other points18 moreDrawn as the hand skeleton in the preview

01

Camera frame

Processed in the browser, never uploaded

02

21 hand points

Google's hand landmark model, inside the gesture recognizer bundle

03

Cursor point

Midpoint of thumb and index tips, mirrored, mapped from the middle of the view

04

One Euro filter

Smooth when the hand is still, quick when it moves

05

Pinch → events

Gap ÷ palm length with hysteresis becomes pointerdown, move, up and click

Try it

Point, pinch, drag. No mouse needed.

Start the camera and hold up one hand. Change the colour, slide the size, drag the sticker, pop eight bubbles against the clock or draw on the pad, all with a pinch.

Live demo

runs in your browser

How to use it. Hold one hand up, palm towards the camera, about an arm's length away. The cursor sits between your thumb and index fingertip.

  • Move: move your hand; the middle of the camera view covers the whole playground.
  • Click: pinch thumb and index together and let go quickly.
  • Drag: pinch, move, then release; it works on the slider, the sticker and the drawing pad.

Browsers don't let a web page move your computer's real mouse, so the hand cursor works inside the playground below. Your mouse works there too.

Colour0 clicks
Size

Drag the sticker · pop the bubbles

✋

The details that make it usable

Raw hand points are jittery. Four small decisions fix that.

01

Cursor between the fingertips

When you pinch, both tips move towards each other, so their midpoint stays put: the click lands where you aimed.

02

Pinch relative to the palm

Dividing the fingertip gap by the palm's length gives the same number whether your hand is close to the camera or far away.

03

Two thresholds, not one

Pinch closes below 0.28 and opens only above 0.42. With one threshold, a hand resting near it would click on and off.

04

Only the middle of the view

The box from 20% to 80% across and 15% to 75% down covers the whole playground, so you never reach for the frame's edge, where tracking is weakest.

The One Euro filter on a simulated hand path

Still, quick move, still. Raise beta and the filter follows fast moves sooner; lower it and it smooths more but lags. The demo uses beta 0.02.

raw fingertipfiltered cursor
250 px485 px720 px0 s1 s2 s3 s4 s5 s

Honest notes

A mouse inside the page, not your computer's.

Browsers don't let websites move the system cursor or click outside the page, for good security reasons. A desktop version would send the same events to the operating system, for example with Python and PyAutoGUI.

What this project is, and isn't

Google's hand model, my cursor, filter, pinch logic and playground, on your device.

Here
Hand modelGoogle's MediaPipe hand landmarks (Apache-2.0), pretrained and used as published
Built hereCursor mapping, One Euro smoothing, pinch with hysteresis, synthetic pointer and click events, and the playground
AccuracyNot benchmarked; thresholds were tuned by hand. Speed depends on your device and is shown live
LimitsOne hand, good light, palm facing the camera. Holding an arm up gets tiring after a minute or two, which is why hand-tracked mice suit short tasks
PrivacyVideo is processed in the browser and never uploaded or recorded

Build notes

How it's built, step by step.

The real code behind this demo, taken from the repository when the site is built. Copy any piece.

In your browser · TypeScript

  1. 1

    Track the hand

    Reuses the gesture recognizer bundle for its 21 hand landmarks, GPU first, CPU if the GPU stalls.

    src/lib/mediapipe.ts:50 · 11 lines

    /** Hand tracking for the cursor: the gesture recognizer already returns the 21 hand landmarks, so no extra model file. */
    export async function loadHandTracker(numHands = 1): Promise<GestureRecognizer> {
      const [{ GestureRecognizer: GR }, vision] = await Promise.all([import("@mediapipe/tasks-vision"), visionFileset()]);
      return withGpuFallback((delegate) =>
        GR.createFromOptions(vision, {
          baseOptions: { modelAssetPath: "/mediapipe/gesture_recognizer.task", delegate },
          runningMode: "VIDEO",
          numHands,
        }),
      );
    }
  2. 2

    Map the hand to the playground

    The comfortable middle of the camera view is stretched over the whole playground, so you never reach for the edges.

    src/lib/virtual-mouse.ts:22 · 5 lines

    export function toArea(p: P, w: number, h: number, box = BOX): P {
      const u = Math.max(0, Math.min(1, (p.x - box.x0) / (box.x1 - box.x0)));
      const v = Math.max(0, Math.min(1, (p.y - box.y0) / (box.y1 - box.y0)));
      return { x: u * w, y: v * h };
    }
  3. 3

    Pinch with hysteresis

    Closes below one threshold and opens only above a higher one, so a hand near the boundary doesn't click on and off.

    src/lib/virtual-mouse.ts:30 · 3 lines

    export function nextPinch(pinched: boolean, ratio: number) {
      return pinched ? ratio < PINCH.up : ratio < PINCH.down;
    }
  4. 4

    Send real pointer events

    Synthetic PointerEvent and MouseEvent objects, dispatched to the element under the cursor so ordinary widgets respond.

    src/components/demos/virtual-mouse-demo.tsx:46 · 5 lines

    /** Synthetic pointer / mouse events, so ordinary page widgets react to the hand cursor. */
    function fire(el: Element, type: string, x: number, y: number, down: boolean) {
      const init = { bubbles: true, cancelable: true, clientX: x, clientY: y, pointerId: 77, pointerType: "mouse", isPrimary: true, button: 0, buttons: down ? 1 : 0 };
      el.dispatchEvent(type === "click" ? new MouseEvent("click", init) : new PointerEvent(type, init));
    }

Notes

The decisions behind it.

Approach

  1. 1

    Reused the self-hosted MediaPipe gesture recognizer for its 21 hand landmarks, GPU first with a CPU fallback.

  2. 2

    Put the cursor at the midpoint of the thumb and index tips (stable while pinching), mirrored it and mapped the comfortable middle of the camera view to the whole playground.

  3. 3

    Smoothed the cursor with a One Euro filter and detected the pinch as fingertip gap over palm length, closing and opening at different thresholds.

  4. 4

    Sent synthetic pointer and click events to whatever sits under the cursor, so ordinary widgets respond, and built a playground to try it.

Honest note

Browsers don't let a web page move the computer's real cursor or click outside the page, so this mouse works inside the playground; a desktop version would send the same events to the operating system. The hand model is Google's; the thresholds were tuned by hand, not benchmarked. It needs one hand in good light, and holding an arm up is tiring after a minute or two.

Reproduce: src/lib/virtual-mouse.ts · src/components/demos/virtual-mouse-demo.tsx

dataset source

Next project

Sign Language Speller