Computer vision
Your facial expression read live from the camera, explained action unit by action unit, with an avatar that copies you.
478
Face points tracked
52
Facial movements read
5
Moods
3.8 MB
Model download
Overview
Faces move in a small number of well-studied ways. Google's Face Landmarker tracks 478 points on the face and scores 52 of those movements every frame, in the browser. This project turns them into something you can see and play with.
The mood is read with rules from the Facial Action Coding System: each expression is a weighted recipe of action units, measured against your own relaxed face after a short calibration. The page shows which movements decided the reading, an avatar mirrors your face, and a challenge asks you to copy five moods as fast as you can.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Camera
your webcam
Frames stay on the device
- 02
Face mesh
478 points
Google's Face Landmarker, GPU first
- 03
Movements
52 blendshapes
Each scored 0–1, every frame
- 04
Calibrate
1.5 s
Your relaxed face becomes the zero
- 05
FACS rules
5 moods
Weighted action units per expression
- 06
Mirror
avatar + challenge
Copies your face; Copy the mood
Inside the model
From a face to a mood.
Google's network scores 52 facial movements; my rules, based on the Facial Action Coding System, turn them into a mood. The faces here are diagrams with example scores.
The model
478 points and 52 facial movements, live.
Google's Face Landmarker turns each camera frame into a 3D face mesh and a score for every facial movement. The mood on top of it is read with rules from the Facial Action Coding System, the standard way psychologists describe expressions.
MediaPipe Face Landmarker
A pretrained model from Google, published for MediaPipe under Apache-2.0 (Google) and used here as it is. It finds one face, places 478 landmarks on it and outputs 52 blendshape scores plus the head's pose. Everything runs in your browser with WebAssembly and your GPU; the model file is self-hosted on this site.
478
face points
52
movements scored
3.8 MB
model download
5
moods read here
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| landmarks | 478 × (x, y, z) | Points on the face mesh: eyes, brows, lips, nose, jaw and the face outline |
| blendshapes | 52 scores, 0–1 | How strongly each facial movement is happening: smile left, brow down right, jaw open… |
| transform | 4×4 matrix | Where the head points: turn, nod and tilt, used to move the avatar |
| mood | 1 of 5 | Computed on this page from the blendshapes with FACS rules, not by Google's model |
01
Camera frame
Read from your webcam, processed in the browser, never uploaded
02
478 face points
Google's face mesh finds the face and tracks it frame to frame
03
52 movements
A second network scores each facial movement from 0 to 1
04
Minus your relaxed face
1.5 seconds of calibration, so your resting face counts as neutral
05
Mood + avatar
FACS rules weigh the action units of each expression; the avatar copies the movements
Try it
Make a face. The mirror reads it.
Mood mirror shows what your expression reads as and why, with an avatar that copies you. Copy the mood challenges you to make five expressions as fast as you can. No camera? The sliders drive the same rules.
Live demo
runs in your browser
Turn on your camera
Video stays on this device: it is processed in your browser and never uploaded or recorded.
No camera? Make faces with sliders. They drive the same mood rules and avatar the camera would.
Your expression reads as
😄 Happy
Why
- Lip corners pulled up AU120.84
- Cheeks raised AU60.60
How moods are read
Each mood is a recipe of facial action units.
FACS, developed by Paul Ekman and Wallace Friesen, codes expressions as action units (AUs): a smile is AU12 pulling the lip corners up, often with AU6 raising the cheeks. Each mood below adds up its action units, weighted, after subtracting your relaxed face.
😄 Happy
- AU12 Lip corners pulled up×1mouthSmileLeft + mouthSmileRight
- AU6 Cheeks raised×0.45cheekSquintLeft + cheekSquintRight
😮 Surprised
- AU1 Inner brows raised×0.6browInnerUp
- AU2 Outer brows raised×0.5browOuterUpLeft + browOuterUpRight
- AU26 Jaw dropped×0.7jawOpen
- AU5 Eyes widened×0.4eyeWideLeft + eyeWideRight
😢 Sad
- AU15 Lip corners pulled down×1mouthFrownLeft + mouthFrownRight
- AU1 Inner brows up, outer not×0.5browInnerUp
- AU17 Chin pushed up×0.35mouthShrugLower
😠 Angry
- AU4 Brows lowered×1browDownLeft + browDownRight
- AU9 Nose wrinkled×0.45noseSneerLeft + noseSneerRight
- AU24 Lips pressed×0.4mouthPressLeft + mouthPressRight
- AU7 Eyelids tightened×0.3eyeSquintLeft + eyeSquintRight
😐 Neutral wins when no recipe has much evidence. Sad and surprised share AU1 (inner brows up), so sadness only counts the inner brows when the outer brows stay down, as in the classic worried brow.
Honest notes
It reads expressions, not feelings.
A smile can be polite and a frown can be concentration. This page shows how facial movements can be measured and combined live; it doesn't know how you feel, and it shouldn't be used to decide anything about anyone.
Google's face model, my rules, avatar and challenge, running on your device.
| Here | |
|---|---|
| Face model | Google's MediaPipe Face Landmarker (Apache-2.0 (Google)), pretrained and used as published; not retrained or re-measured here |
| Built here | FACS mood rules, relaxed-face calibration, smoothing, the avatar, Copy the mood and the slider mode |
| Accuracy | Not measured: the rule weights were set by hand from the FACS definitions and checked on synthetic expressions, not on a labelled set of real faces |
| Privacy | Video is processed in your browser and never uploaded or recorded; the challenge recap shows the avatar, not a photo |
| Limits | Works best facing the camera in good light; side views, strong shadows, hands on the face or heavy glasses weaken the readings |
Build notes
How it's built, step by step.
The real code behind this demo, taken from the repository when the site is built. Copy any piece.
In your browser · TypeScript
- 1
Load Google's face model
MediaPipe's Face Landmarker in video mode, asked for blendshapes and the head's transformation matrix; GPU first, CPU if the GPU stalls.
src/lib/mediapipe.ts:36 · 13 lines
/** Google's Face Landmarker in video mode, self-hosted; blendshapes and head pose only when a demo needs them. */ export async function loadFaceLandmarker(extras = true): Promise<FaceLandmarker> { const [{ FaceLandmarker: FL }, vision] = await Promise.all([import("@mediapipe/tasks-vision"), visionFileset()]); return withGpuFallback((delegate) => FL.createFromOptions(vision, { baseOptions: { modelAssetPath: "/mediapipe/face_landmarker.task", delegate }, runningMode: "VIDEO", numFaces: 1, outputFaceBlendshapes: extras, outputFacialTransformationMatrixes: extras, }), ); } - 2
Weigh the action units of each mood
Subtracts the relaxed face, scales each blendshape group and takes a weighted mean per expression, keeping every unit's share for the 'Why' list.
src/lib/face-mood.ts:39 · 13 lines
/** Evidence (0–1) for each expression and how much each action unit contributed. */ export function evidence(s: Shapes, baseline: Shapes = {}) { const at = (n: string) => Math.max(0, (s[n] ?? 0) - (baseline[n] ?? 0)); const mean = (ns: string[]) => ns.reduce((a, n) => a + at(n), 0) / ns.length; const out = {} as Record<Exclude<Mood, "neutral">, { e: number; parts: { au: string; label: string; v: number; share: number }[] }>; for (const [mood, parts] of Object.entries(RULES) as [Exclude<Mood, "neutral">, Part[]][]) { const wsum = parts.reduce((a, p) => a + p.w, 0); const vals = parts.map((p) => clamp01((mean(p.shapes) - (p.minus ? mean(p.minus) : 0)) * p.gain)); const e = parts.reduce((a, p, i) => a + p.w * vals[i], 0) / wsum; out[mood] = { e, parts: parts.map((p, i) => ({ au: p.au, label: p.label, v: vals[i], share: (p.w * vals[i]) / wsum })) }; } return out; } - 3
Turn evidence into five probabilities
Neutral gets the leftover when no recipe is strong; a sharpened softmax gives the bars and the top mood.
src/lib/face-mood.ts:53 · 11 lines
/** Probabilities over the five moods. Neutral wins when no expression has much evidence. */ export function moodProbs(s: Shapes, baseline: Shapes = {}) { const ev = evidence(s, baseline); const strongest = Math.max(...Object.values(ev).map((x) => x.e)); const score: Record<Mood, number> = { happy: ev.happy.e, surprised: ev.surprised.e, sad: ev.sad.e, angry: ev.angry.e, neutral: clamp01(0.45 - strongest) }; const ex = MOODS.map((m) => Math.exp(SHARP * score[m])); const sum = ex.reduce((a, v) => a + v, 0); const probs = Object.fromEntries(MOODS.map((m, i) => [m, ex[i] / sum])) as Record<Mood, number>; const top = MOODS.reduce((a, m) => (probs[m] > probs[a] ? m : a), "neutral" as Mood); return { probs, top, ev }; } - 4
Read the head's turn, nod and tilt
Euler angles from MediaPipe's 4×4 transformation matrix, used to rotate and shift the avatar's features.
src/lib/face-mood.ts:80 · 6 lines
/** Head yaw / pitch / roll in degrees from MediaPipe's 4×4 facial transformation matrix (column-major). */ export function headPose(m: ArrayLike<number> | undefined) { if (!m || m.length < 16) return { yaw: 0, pitch: 0, roll: 0 }; const deg = 180 / Math.PI; return { yaw: Math.asin(Math.max(-1, Math.min(1, -m[2]))) * deg, pitch: Math.atan2(m[6], m[10]) * deg, roll: Math.atan2(m[1], m[0]) * deg }; } - 5
Draw the mirror avatar
An SVG face whose eyes, brows, cheeks and mouth follow the blendshapes, mirrored like a selfie.
src/components/demos/face-mood-demo.tsx:48 · 61 lines
/** * An original cartoon face driven by the blendshapes, mirrored like a selfie. * MediaPipe names sides from the camera's point of view, so "Left" shapes move the avatar's right side on screen. */ export function Avatar({ s, pose, size = 220 }: { s: Shapes; pose: Pose; size?: number }) { const g = (n: string) => s[n] ?? 0; const dx = Math.max(-16, Math.min(16, -pose.yaw * 0.8)); const dy = Math.max(-10, Math.min(10, pose.pitch * 0.5)); const roll = Math.max(-25, Math.min(25, -pose.roll)); const smile = (g("mouthSmileLeft") + g("mouthSmileRight")) / 2; const frown = (g("mouthFrownLeft") + g("mouthFrownRight")) / 2; const jaw = g("jawOpen"); const pucker = g("mouthPucker"); const browUp = g("browInnerUp"); const eye = (side: "L" | "R") => { const k = side === "L" ? "Right" : "Left"; // screen-left eye follows MediaPipe's "Right" shapes
Notes
The decisions behind it.
Approach
- 1
Ran Google's Face Landmarker in the browser with MediaPipe, self-hosted, GPU first with a CPU fallback, asking for blendshapes and the head's pose.
- 2
Wrote the mood rules from FACS: happy (AU12, AU6), surprised (AU1, AU2, AU5, AU26), sad (AU15, AU1 without AU2, AU17) and angry (AU4, AU9, AU24, AU7), each a weighted mean of its action units.
- 3
Subtracted the visitor's relaxed face, averaged over 1.5 seconds, and smoothed the blendshapes between frames so readings don't flicker.
- 4
Drew an original avatar driven by the same blendshapes and head pose, built the Copy the mood challenge on top, and added sliders for visitors without a camera.
Honest note
The face model is Google's, used as published. The mood rules are mine and have not been measured on a labelled set of faces: the weights were set by hand from the FACS definitions and checked on synthetic expressions. Above all it reads expressions, not feelings: a smile can be polite, a frown can be concentration, and expressions vary between people and cultures, so it should never be used to judge anyone.
Reproduce: src/lib/face-mood.ts · src/components/demos/face-mood-demo.tsx
dataset sourceNext project
Drowsiness Detector