←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

Hands-free typing for accessibility: the head or the eyes point, a blink presses, and a scanning mode works with blinks alone.

3

Ways to select

head · eyes

Pointers

2

Suggestion languages

none

Extra download

Overview

For people living with ALS, a spinal cord injury or cerebral palsy, a keyboard can be out of reach while a small head movement or a blink is not. Augmentative and alternative communication (AAC) devices turn those movements into words, usually with dedicated eye-tracking hardware.

This project does it with a webcam. Google's face model gives the head's pose and the eyelid points every frame; the page turns head movement into a pointer, a deliberate blink or a dwell into a key press, and offers row/column scanning for people who can only blink. Word suggestions in English and Bahasa Indonesia and a Speak button complete the message.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Camera

    your webcam

    Frames stay on the device

  2. 02

    Face model

    478 points + pose

    Google's Face Landmarker

  3. 03

    Calibrate

    1.5 s

    Neutral head pose and open-eye EAR

  4. 04

    Point

    head or eyes

    Head pose or iris gaze, smoothed

  5. 05

    Press

    blink · dwell · scan

    Three methods for different bodies

  6. 06

    Speak

    EN · ID

    Word suggestions and text-to-speech

Inside the model

From a head turn to a word.

Google's network gives the head's pose, the eyelids and the irises; the pointer, clicks and keyboard are my code. The faces are diagrams; the settings are the demo's.

Face
FACE DETECTOR: A FACE IS HERE

The signals

Your head or eyes as a pointer, a blink as the button.

People with ALS, spinal cord injuries or cerebral palsy often can't use a keyboard or mouse but can move their head or blink. Augmentative and alternative communication (AAC) tools turn those small movements into words. This one runs on a webcam.

MediaPipe Face Landmarker

Google's pretrained face model (Apache-2.0 (Google)), the same one behind Face Mood Mirror and the Drowsiness Detector. It gives the head's pose every frame and the eyelid points for the eye aspect ratio; the pointer, blink detection, dwell, scanning and word suggestions are written for this page.

3

select methods

≥ 0.45 s

deliberate blink

2

suggestion languages

3.8 MB

model download

Model page

Column dictionary

● used by the model

ColumnTypeWhat it means
head poseyaw · pitch · rollFrom the face model's 4×4 transformation matrix: where your head points
eye points2 × 6Eyelid points for the eye aspect ratio: when the eyes close, how long they stay closed
iris points2 × 5Iris centres and edges: where the eyes look, for the eye pointer
other points456Tracked by the model, not needed here

01

Camera frame

Processed in the browser, never uploaded

02

Face model

478 points and the head's pose, every frame

03

Head or eyes → pointer

Head turn around your neutral pose, or iris position after a 5-dot eye calibration, smoothed

04

Blink or dwell

An instant blink, a held blink, or holding still on a key presses it

05

Words → voice

Word suggestions speed typing; Speak reads the message aloud

Try it

Type a message without your hands.

Start the camera and look at the middle of the keyboard to calibrate. Point with your head, or switch to Eyes and follow the five dots, then blink to type. Try Blink only to type with blinks alone, and Speak to hear it. No camera? The arrow keys and Space do the same.

Live demo

runs in your browser

Select with

Point with

Blink click

Word suggestions

Instant: every blink clicks the moment your eyes close (one click per 0.35 s at most). Natural blinks click too, so keep suggestions and delete handy.

Your message

I need water

No camera? Click the keyboard, then use the arrow keys to move and Space as the blink. Keys can also be clicked.

Three ways to press a key

Different bodies need different inputs.

Point + blink

With the head or the eyes

Point with small head movements, or with your eyes after a 5-dot calibration. Instant mode clicks the moment your eyes close (at most once per 0.35 s); Hold mode needs about 0.45 s, so natural blinks don't type.

Point + dwell

For people who can't blink on purpose

Point with your head or eyes and stay on a key until its bar fills. No blinking needed; the dwell time is adjustable.

Blink only (scanning)

For people who can't move their head

Rows light up one after another; blink to pick a row, then keys light up; blink again to type. The classic switch-access method used in AAC devices.

The keyboard is alphabetical, as many AAC boards are, so scanning is predictable. The top row suggests the next word from short, hand-made lists of everyday words in English and Bahasa Indonesia (needs, feelings, people, places), because every key saved matters when each press takes a second or two.

Honest notes

A demonstration of the idea, not a medical device.

Dedicated AAC devices use infrared eye trackers, are fitted by therapists and are tested with their users. This page shows how far an ordinary webcam can go, and has not been tested with people who rely on AAC.

What this project is, and isn't

Google's face model, my pointer, blink, dwell, scanning and suggestions, all on your device.

Here
Face modelGoogle's MediaPipe Face Landmarker (Apache-2.0 (Google)), pretrained and used as published
Built hereHead and eye pointers with calibration and smoothing, instant and held blink clicks, dwell, row/column scanning, word suggestions, speech and the keyboard
Eye pointerWebcam gaze is rough: iris position relative to the eye corners and lids, mapped by a 5-dot calibration. It assumes a still head; big keys make it usable, a dedicated eye tracker is far more precise
Instant blinkFast but unforgiving: natural blinks click too. Hold mode, dwell or scanning avoid that
SpeedExpect a few words per minute, typical for head and blink input; suggestions help most
LimitsNeeds a lit, visible face; glasses glare, strong head tremor or drooping eyelids need different settings (flip, range, dwell, scan speed)
PrivacyVideo is processed in the browser and never uploaded or recorded; the message stays on the page

Build notes

How it's built, step by step.

The real code behind this demo, taken from the repository when the site is built. Copy any piece.

In your browser · TypeScript

  1. 1

    Head pose to pointer

    Yaw and pitch relative to the calibrated neutral pose, scaled by the chosen range and mirrored like a selfie.

    src/lib/head-keyboard.ts:9 · 7 lines

    /** Head turn → pointer in 0–1, around the calibrated neutral pose, mirrored like a selfie. */
    export function headPointer(p: Pose, c: Calib, range: Range, flip = { x: false, y: false }) {
      const dx = (p.yaw - c.yaw) / range.yaw;
      const dy = (p.pitch - c.pitch) / range.pitch;
      const clamp = (v: number) => Math.max(0, Math.min(1, v));
      return { x: clamp(0.5 + (flip.x ? dx : -dx) / 2), y: clamp(0.5 + (flip.y ? -dy : dy) / 2) };
    }
  2. 2

    Where the eyes look

    Iris centre between the eye corners (left–right) and between the lids (up–down), averaged over both eyes.

    src/lib/head-keyboard.ts:55 · 18 lines

    /**
     * Where the eyes look, as two ratios averaged over both eyes: horizontal = iris position between the eye corners,
     * vertical = iris position between the upper and lower lid. Relative to the eye, so small head movements matter less.
     */
    export function gazeRatios(pts: Pt[]) {
      let h = 0;
      let v = 0;
      for (const e of EYES) {
        const i = pts[e.iris];
        const a = pts[e.c1];
        const b = pts[e.c2];
        const top = (pts[e.top[0]].y + pts[e.top[1]].y) / 2;
        const bottom = (pts[e.bottom[0]].y + pts[e.bottom[1]].y) / 2;
        h += (i.x - a.x) / (b.x - a.x || 1e-6);
        v += (i.y - top) / (bottom - top || 1e-6);
      }
      return { h: h / EYES.length, v: v / EYES.length };
    }
  3. 3

    Gaze to pointer, from a 5-dot calibration

    Linear interpolation between the gaze measured at the left/right and top/bottom dots; the calibration also fixes the direction.

    src/lib/head-keyboard.ts:89 · 8 lines

    /** Gaze ratios → pointer in 0–1, by linear interpolation between the calibrated targets (direction comes from calibration). */
    export function gazePointer(g: { h: number; v: number }, m: GazeMap) {
      const clamp = (x: number) => Math.max(0, Math.min(1, x));
      const T = GAZE_TARGETS;
      const u = (g.h - m.hL) / (m.hR - m.hL || 1e-6);
      const w = (g.v - m.vT) / (m.vB - m.vT || 1e-6);
      return { x: clamp(T[1].x + u * (T[2].x - T[1].x)), y: clamp(T[3].y + w * (T[4].y - T[3].y)) };
    }
  4. 4

    Instant blink click

    Clicks on the open-to-closed edge, at most once per 0.35 s, so a blink presses immediately.

    src/lib/head-keyboard.ts:40 · 5 lines

    export function instantBlinkStep(s: InstantState, t: number, earNow: number, openEar: number, ratio = BLINK.ratio, refractoryMs = INSTANT.refractoryMs) {
      const closed = earNow < openEar * ratio;
      const click = closed && !s.closed && t - s.lastClick >= refractoryMs;
      return { state: { closed, lastClick: click ? t : s.lastClick }, closed, click };
    }
  5. 5

    A held blink, for fewer accidental clicks

    Both eyes closed for 0.45 to 2.5 seconds, firing when they reopen: natural blinks and resting the eyes don't type.

    src/lib/head-keyboard.ts:20 · 13 lines

    /**
     * A deliberate blink: both eyes closed (EAR below a share of your open-eye EAR) for at least `minMs` and at most
     * `maxMs`, then opened. Natural blinks (about 0.1–0.3 s) and resting your eyes are ignored. Fires on reopening.
     */
    export function blinkStep(s: BlinkState, t: number, earNow: number, openEar: number, cfg = BLINK) {
      const closed = earNow < openEar * cfg.ratio;
      if (closed) {
        const since = s.closedSince ?? t;
        return { state: { closedSince: since }, closed, progress: Math.min(1, (t - since) / cfg.minMs), click: false };
      }
      const dur = s.closedSince === null ? 0 : t - s.closedSince;
      return { state: { closedSince: null }, closed, progress: 0, click: dur >= cfg.minMs && dur <= cfg.maxMs };
    }
  6. 6

    Dwell to press

    Staying on one key for the dwell time presses it once; moving to another key starts again.

    src/lib/head-keyboard.ts:99 · 7 lines

    /** Dwell selection: staying on one key for `ms` selects it once; move to another key to select again. */
    export function dwellStep(d: Dwell, t: number, key: string | null, ms: number) {
      if (key !== d.key) return { state: { key, since: t, done: false }, progress: 0, click: false };
      if (!key || d.done) return { state: d, progress: d.done ? 1 : 0, click: false };
      const progress = Math.min(1, (t - d.since) / ms);
      return { state: { ...d, done: progress >= 1 }, progress, click: progress >= 1 };
    }
  7. 7

    Row/column scanning with blinks alone

    Rows light up in turn, a blink picks the row, keys light up, a blink types; after the last key it returns to the rows.

    src/lib/head-keyboard.ts:109 · 11 lines

    export function scanStep(s: Scan, t: number, rows: number[], stepMs: number, select: boolean): { state: Scan; pick: { row: number; col: number } | null } {
      if (select) {
        if (s.level === "row") return { state: { level: "key", row: s.row, col: 0, last: t }, pick: null };
        return { state: { level: "row", row: 0, col: 0, last: t }, pick: { row: s.row, col: s.col } };
      }
      if (t - s.last < stepMs) return { state: s, pick: null };
      if (s.level === "row") return { state: { ...s, row: (s.row + 1) % rows.length, last: t }, pick: null };
      // after the last key of a row, go back to row scanning, so a missed row costs one pass
      if (s.col + 1 >= rows[s.row]) return { state: { level: "row", row: s.row, col: 0, last: t }, pick: null };
      return { state: { ...s, col: s.col + 1, last: t }, pick: null };
    }
  8. 8

    Word suggestions

    The most frequent words that start with what you are typing, from short hand-made lists in English and Bahasa Indonesia.

    src/lib/head-keyboard.ts:121 · 7 lines

    /** Word completion from a frequency-ordered list: the first words that start with what you are typing. */
    export function predict(text: string, words: string[], n = 3) {
      const m = text.match(/([a-zA-Z']+)$/);
      const prefix = (m?.[1] ?? "").toLowerCase();
      if (!prefix) return words.slice(0, n);
      return words.filter((w) => w.startsWith(prefix) && w !== prefix).slice(0, n);
    }

Notes

The decisions behind it.

Approach

  1. 1

    Reused the self-hosted Face Landmarker for the head's transformation matrix and the eyelid points, and calibrated the neutral pose and open-eye EAR for each person.

  2. 2

    Mapped head turn and nod to a pointer over the keyboard, smoothed with a One Euro filter, with adjustable range and flips.

  3. 3

    Added an eye pointer from the iris landmarks (iris between the eye corners and lids), mapped by a 5-dot calibration and frozen during blinks, with clicks placed where the eyes were just before the lids moved.

  4. 4

    Offered an instant blink click (on the closing edge, one per 0.35 s) and a held blink (0.45 to 2.5 s, firing on reopening), plus dwell selection and row/column scanning.

  5. 5

    Built an alphabetical keyboard with word suggestions in English and Bahasa Indonesia, text-to-speech, and arrow keys plus Space as a fallback and switch input.

Honest note

A demonstration, not a medical device: dedicated AAC systems use infrared eye trackers and are fitted and tested with their users, and this page hasn't been tested with people who rely on AAC. Expect a few words per minute. It needs a lit, visible face; glare from glasses, tremor or drooping eyelids need adjusted settings. The face model is Google's; everything on top is mine.

Reproduce: src/lib/head-keyboard.ts · src/components/demos/head-keyboard-demo.tsx

dataset source

Next project

Air Synth