Computer vision
Hands-free typing for accessibility: the head or the eyes point, a blink presses, and a scanning mode works with blinks alone.
3
Ways to select
head · eyes
Pointers
2
Suggestion languages
none
Extra download
Overview
For people living with ALS, a spinal cord injury or cerebral palsy, a keyboard can be out of reach while a small head movement or a blink is not. Augmentative and alternative communication (AAC) devices turn those movements into words, usually with dedicated eye-tracking hardware.
This project does it with a webcam. Google's face model gives the head's pose and the eyelid points every frame; the page turns head movement into a pointer, a deliberate blink or a dwell into a key press, and offers row/column scanning for people who can only blink. Word suggestions in English and Bahasa Indonesia and a Speak button complete the message.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Camera
your webcam
Frames stay on the device
- 02
Face model
478 points + pose
Google's Face Landmarker
- 03
Calibrate
1.5 s
Neutral head pose and open-eye EAR
- 04
Point
head or eyes
Head pose or iris gaze, smoothed
- 05
Press
blink · dwell · scan
Three methods for different bodies
- 06
Speak
EN · ID
Word suggestions and text-to-speech
Inside the model
From a head turn to a word.
Google's network gives the head's pose, the eyelids and the irises; the pointer, clicks and keyboard are my code. The faces are diagrams; the settings are the demo's.
The signals
Your head or eyes as a pointer, a blink as the button.
People with ALS, spinal cord injuries or cerebral palsy often can't use a keyboard or mouse but can move their head or blink. Augmentative and alternative communication (AAC) tools turn those small movements into words. This one runs on a webcam.
MediaPipe Face Landmarker
Google's pretrained face model (Apache-2.0 (Google)), the same one behind Face Mood Mirror and the Drowsiness Detector. It gives the head's pose every frame and the eyelid points for the eye aspect ratio; the pointer, blink detection, dwell, scanning and word suggestions are written for this page.
3
select methods
≥ 0.45 s
deliberate blink
2
suggestion languages
3.8 MB
model download
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| head pose | yaw · pitch · roll | From the face model's 4×4 transformation matrix: where your head points |
| eye points | 2 × 6 | Eyelid points for the eye aspect ratio: when the eyes close, how long they stay closed |
| iris points | 2 × 5 | Iris centres and edges: where the eyes look, for the eye pointer |
| other points | 456 | Tracked by the model, not needed here |
01
Camera frame
Processed in the browser, never uploaded
02
Face model
478 points and the head's pose, every frame
03
Head or eyes → pointer
Head turn around your neutral pose, or iris position after a 5-dot eye calibration, smoothed
04
Blink or dwell
An instant blink, a held blink, or holding still on a key presses it
05
Words → voice
Word suggestions speed typing; Speak reads the message aloud
Try it
Type a message without your hands.
Start the camera and look at the middle of the keyboard to calibrate. Point with your head, or switch to Eyes and follow the five dots, then blink to type. Try Blink only to type with blinks alone, and Speak to hear it. No camera? The arrow keys and Space do the same.
Live demo
runs in your browser
Select with
Point with
Blink click
Word suggestions
Instant: every blink clicks the moment your eyes close (one click per 0.35 s at most). Natural blinks click too, so keep suggestions and delete handy.
Your message
I need water
No camera? Click the keyboard, then use the arrow keys to move and Space as the blink. Keys can also be clicked.
Three ways to press a key
Different bodies need different inputs.
Point + blink
With the head or the eyes
Point with small head movements, or with your eyes after a 5-dot calibration. Instant mode clicks the moment your eyes close (at most once per 0.35 s); Hold mode needs about 0.45 s, so natural blinks don't type.
Point + dwell
For people who can't blink on purpose
Point with your head or eyes and stay on a key until its bar fills. No blinking needed; the dwell time is adjustable.
Blink only (scanning)
For people who can't move their head
Rows light up one after another; blink to pick a row, then keys light up; blink again to type. The classic switch-access method used in AAC devices.
The keyboard is alphabetical, as many AAC boards are, so scanning is predictable. The top row suggests the next word from short, hand-made lists of everyday words in English and Bahasa Indonesia (needs, feelings, people, places), because every key saved matters when each press takes a second or two.
Honest notes
A demonstration of the idea, not a medical device.
Dedicated AAC devices use infrared eye trackers, are fitted by therapists and are tested with their users. This page shows how far an ordinary webcam can go, and has not been tested with people who rely on AAC.
Google's face model, my pointer, blink, dwell, scanning and suggestions, all on your device.
| Here | |
|---|---|
| Face model | Google's MediaPipe Face Landmarker (Apache-2.0 (Google)), pretrained and used as published |
| Built here | Head and eye pointers with calibration and smoothing, instant and held blink clicks, dwell, row/column scanning, word suggestions, speech and the keyboard |
| Eye pointer | Webcam gaze is rough: iris position relative to the eye corners and lids, mapped by a 5-dot calibration. It assumes a still head; big keys make it usable, a dedicated eye tracker is far more precise |
| Instant blink | Fast but unforgiving: natural blinks click too. Hold mode, dwell or scanning avoid that |
| Speed | Expect a few words per minute, typical for head and blink input; suggestions help most |
| Limits | Needs a lit, visible face; glasses glare, strong head tremor or drooping eyelids need different settings (flip, range, dwell, scan speed) |
| Privacy | Video is processed in the browser and never uploaded or recorded; the message stays on the page |
Build notes
How it's built, step by step.
The real code behind this demo, taken from the repository when the site is built. Copy any piece.
In your browser · TypeScript
- 1
Head pose to pointer
Yaw and pitch relative to the calibrated neutral pose, scaled by the chosen range and mirrored like a selfie.
src/lib/head-keyboard.ts:9 · 7 lines
/** Head turn → pointer in 0–1, around the calibrated neutral pose, mirrored like a selfie. */ export function headPointer(p: Pose, c: Calib, range: Range, flip = { x: false, y: false }) { const dx = (p.yaw - c.yaw) / range.yaw; const dy = (p.pitch - c.pitch) / range.pitch; const clamp = (v: number) => Math.max(0, Math.min(1, v)); return { x: clamp(0.5 + (flip.x ? dx : -dx) / 2), y: clamp(0.5 + (flip.y ? -dy : dy) / 2) }; } - 2
Where the eyes look
Iris centre between the eye corners (left–right) and between the lids (up–down), averaged over both eyes.
src/lib/head-keyboard.ts:55 · 18 lines
/** * Where the eyes look, as two ratios averaged over both eyes: horizontal = iris position between the eye corners, * vertical = iris position between the upper and lower lid. Relative to the eye, so small head movements matter less. */ export function gazeRatios(pts: Pt[]) { let h = 0; let v = 0; for (const e of EYES) { const i = pts[e.iris]; const a = pts[e.c1]; const b = pts[e.c2]; const top = (pts[e.top[0]].y + pts[e.top[1]].y) / 2; const bottom = (pts[e.bottom[0]].y + pts[e.bottom[1]].y) / 2; h += (i.x - a.x) / (b.x - a.x || 1e-6); v += (i.y - top) / (bottom - top || 1e-6); } return { h: h / EYES.length, v: v / EYES.length }; } - 3
Gaze to pointer, from a 5-dot calibration
Linear interpolation between the gaze measured at the left/right and top/bottom dots; the calibration also fixes the direction.
src/lib/head-keyboard.ts:89 · 8 lines
/** Gaze ratios → pointer in 0–1, by linear interpolation between the calibrated targets (direction comes from calibration). */ export function gazePointer(g: { h: number; v: number }, m: GazeMap) { const clamp = (x: number) => Math.max(0, Math.min(1, x)); const T = GAZE_TARGETS; const u = (g.h - m.hL) / (m.hR - m.hL || 1e-6); const w = (g.v - m.vT) / (m.vB - m.vT || 1e-6); return { x: clamp(T[1].x + u * (T[2].x - T[1].x)), y: clamp(T[3].y + w * (T[4].y - T[3].y)) }; } - 4
Instant blink click
Clicks on the open-to-closed edge, at most once per 0.35 s, so a blink presses immediately.
src/lib/head-keyboard.ts:40 · 5 lines
export function instantBlinkStep(s: InstantState, t: number, earNow: number, openEar: number, ratio = BLINK.ratio, refractoryMs = INSTANT.refractoryMs) { const closed = earNow < openEar * ratio; const click = closed && !s.closed && t - s.lastClick >= refractoryMs; return { state: { closed, lastClick: click ? t : s.lastClick }, closed, click }; } - 5
A held blink, for fewer accidental clicks
Both eyes closed for 0.45 to 2.5 seconds, firing when they reopen: natural blinks and resting the eyes don't type.
src/lib/head-keyboard.ts:20 · 13 lines
/** * A deliberate blink: both eyes closed (EAR below a share of your open-eye EAR) for at least `minMs` and at most * `maxMs`, then opened. Natural blinks (about 0.1–0.3 s) and resting your eyes are ignored. Fires on reopening. */ export function blinkStep(s: BlinkState, t: number, earNow: number, openEar: number, cfg = BLINK) { const closed = earNow < openEar * cfg.ratio; if (closed) { const since = s.closedSince ?? t; return { state: { closedSince: since }, closed, progress: Math.min(1, (t - since) / cfg.minMs), click: false }; } const dur = s.closedSince === null ? 0 : t - s.closedSince; return { state: { closedSince: null }, closed, progress: 0, click: dur >= cfg.minMs && dur <= cfg.maxMs }; } - 6
Dwell to press
Staying on one key for the dwell time presses it once; moving to another key starts again.
src/lib/head-keyboard.ts:99 · 7 lines
/** Dwell selection: staying on one key for `ms` selects it once; move to another key to select again. */ export function dwellStep(d: Dwell, t: number, key: string | null, ms: number) { if (key !== d.key) return { state: { key, since: t, done: false }, progress: 0, click: false }; if (!key || d.done) return { state: d, progress: d.done ? 1 : 0, click: false }; const progress = Math.min(1, (t - d.since) / ms); return { state: { ...d, done: progress >= 1 }, progress, click: progress >= 1 }; } - 7
Row/column scanning with blinks alone
Rows light up in turn, a blink picks the row, keys light up, a blink types; after the last key it returns to the rows.
src/lib/head-keyboard.ts:109 · 11 lines
export function scanStep(s: Scan, t: number, rows: number[], stepMs: number, select: boolean): { state: Scan; pick: { row: number; col: number } | null } { if (select) { if (s.level === "row") return { state: { level: "key", row: s.row, col: 0, last: t }, pick: null }; return { state: { level: "row", row: 0, col: 0, last: t }, pick: { row: s.row, col: s.col } }; } if (t - s.last < stepMs) return { state: s, pick: null }; if (s.level === "row") return { state: { ...s, row: (s.row + 1) % rows.length, last: t }, pick: null }; // after the last key of a row, go back to row scanning, so a missed row costs one pass if (s.col + 1 >= rows[s.row]) return { state: { level: "row", row: s.row, col: 0, last: t }, pick: null }; return { state: { ...s, col: s.col + 1, last: t }, pick: null }; } - 8
Word suggestions
The most frequent words that start with what you are typing, from short hand-made lists in English and Bahasa Indonesia.
src/lib/head-keyboard.ts:121 · 7 lines
/** Word completion from a frequency-ordered list: the first words that start with what you are typing. */ export function predict(text: string, words: string[], n = 3) { const m = text.match(/([a-zA-Z']+)$/); const prefix = (m?.[1] ?? "").toLowerCase(); if (!prefix) return words.slice(0, n); return words.filter((w) => w.startsWith(prefix) && w !== prefix).slice(0, n); }
Notes
The decisions behind it.
Approach
- 1
Reused the self-hosted Face Landmarker for the head's transformation matrix and the eyelid points, and calibrated the neutral pose and open-eye EAR for each person.
- 2
Mapped head turn and nod to a pointer over the keyboard, smoothed with a One Euro filter, with adjustable range and flips.
- 3
Added an eye pointer from the iris landmarks (iris between the eye corners and lids), mapped by a 5-dot calibration and frozen during blinks, with clicks placed where the eyes were just before the lids moved.
- 4
Offered an instant blink click (on the closing edge, one per 0.35 s) and a held blink (0.45 to 2.5 s, firing on reopening), plus dwell selection and row/column scanning.
- 5
Built an alphabetical keyboard with word suggestions in English and Bahasa Indonesia, text-to-speech, and arrow keys plus Space as a fallback and switch input.
Honest note
A demonstration, not a medical device: dedicated AAC systems use infrared eye trackers and are fitted and tested with their users, and this page hasn't been tested with people who rely on AAC. Expect a few words per minute. It needs a lit, visible face; glare from glasses, tremor or drooping eyelids need adjusted settings. The face model is Google's; everything on top is mine.
Reproduce: src/lib/head-keyboard.ts · src/components/demos/head-keyboard-demo.tsx
dataset sourceNext project
Air Synth