←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

Your facial expression read live from the camera, explained action unit by action unit, with an avatar that copies you.

478

Face points tracked

52

Facial movements read

5

Moods

3.8 MB

Model download

Overview

Faces move in a small number of well-studied ways. Google's Face Landmarker tracks 478 points on the face and scores 52 of those movements every frame, in the browser. This project turns them into something you can see and play with.

The mood is read with rules from the Facial Action Coding System: each expression is a weighted recipe of action units, measured against your own relaxed face after a short calibration. The page shows which movements decided the reading, an avatar mirrors your face, and a challenge asks you to copy five moods as fast as you can.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Camera

    your webcam

    Frames stay on the device

  2. 02

    Face mesh

    478 points

    Google's Face Landmarker, GPU first

  3. 03

    Movements

    52 blendshapes

    Each scored 0–1, every frame

  4. 04

    Calibrate

    1.5 s

    Your relaxed face becomes the zero

  5. 05

    FACS rules

    5 moods

    Weighted action units per expression

  6. 06

    Mirror

    avatar + challenge

    Copies your face; Copy the mood

Inside the model

From a face to a mood.

Google's network scores 52 facial movements; my rules, based on the Facial Action Coding System, turn them into a mood. The faces here are diagrams with example scores.

Face
FACE DETECTOR: A FACE IS HERE

The model

478 points and 52 facial movements, live.

Google's Face Landmarker turns each camera frame into a 3D face mesh and a score for every facial movement. The mood on top of it is read with rules from the Facial Action Coding System, the standard way psychologists describe expressions.

MediaPipe Face Landmarker

A pretrained model from Google, published for MediaPipe under Apache-2.0 (Google) and used here as it is. It finds one face, places 478 landmarks on it and outputs 52 blendshape scores plus the head's pose. Everything runs in your browser with WebAssembly and your GPU; the model file is self-hosted on this site.

478

face points

52

movements scored

3.8 MB

model download

5

moods read here

Model page

Column dictionary

● used by the model

ColumnTypeWhat it means
landmarks478 × (x, y, z)Points on the face mesh: eyes, brows, lips, nose, jaw and the face outline
blendshapes52 scores, 0–1How strongly each facial movement is happening: smile left, brow down right, jaw open…
transform4×4 matrixWhere the head points: turn, nod and tilt, used to move the avatar
mood1 of 5Computed on this page from the blendshapes with FACS rules, not by Google's model

01

Camera frame

Read from your webcam, processed in the browser, never uploaded

02

478 face points

Google's face mesh finds the face and tracks it frame to frame

03

52 movements

A second network scores each facial movement from 0 to 1

04

Minus your relaxed face

1.5 seconds of calibration, so your resting face counts as neutral

05

Mood + avatar

FACS rules weigh the action units of each expression; the avatar copies the movements

Try it

Make a face. The mirror reads it.

Mood mirror shows what your expression reads as and why, with an avatar that copies you. Copy the mood challenges you to make five expressions as fast as you can. No camera? The sliders drive the same rules.

Live demo

runs in your browser

Turn on your camera

Video stays on this device: it is processed in your browser and never uploaded or recorded.

No camera? Make faces with sliders. They drive the same mood rules and avatar the camera would.

Your expression reads as

😄 Happy

😄 Happy98%
😮 Surprised1%
😢 Sad0%
😠 Angry0%
😐 Neutral0%

Why

  • Lip corners pulled up AU120.84
  • Cheeks raised AU60.60

How moods are read

Each mood is a recipe of facial action units.

FACS, developed by Paul Ekman and Wallace Friesen, codes expressions as action units (AUs): a smile is AU12 pulling the lip corners up, often with AU6 raising the cheeks. Each mood below adds up its action units, weighted, after subtracting your relaxed face.

😄 Happy

  • AU12 Lip corners pulled up×1mouthSmileLeft + mouthSmileRight
  • AU6 Cheeks raised×0.45cheekSquintLeft + cheekSquintRight

😮 Surprised

  • AU1 Inner brows raised×0.6browInnerUp
  • AU2 Outer brows raised×0.5browOuterUpLeft + browOuterUpRight
  • AU26 Jaw dropped×0.7jawOpen
  • AU5 Eyes widened×0.4eyeWideLeft + eyeWideRight

😢 Sad

  • AU15 Lip corners pulled down×1mouthFrownLeft + mouthFrownRight
  • AU1 Inner brows up, outer not×0.5browInnerUp
  • AU17 Chin pushed up×0.35mouthShrugLower

😠 Angry

  • AU4 Brows lowered×1browDownLeft + browDownRight
  • AU9 Nose wrinkled×0.45noseSneerLeft + noseSneerRight
  • AU24 Lips pressed×0.4mouthPressLeft + mouthPressRight
  • AU7 Eyelids tightened×0.3eyeSquintLeft + eyeSquintRight

😐 Neutral wins when no recipe has much evidence. Sad and surprised share AU1 (inner brows up), so sadness only counts the inner brows when the outer brows stay down, as in the classic worried brow.

Honest notes

It reads expressions, not feelings.

A smile can be polite and a frown can be concentration. This page shows how facial movements can be measured and combined live; it doesn't know how you feel, and it shouldn't be used to decide anything about anyone.

What this project is, and isn't

Google's face model, my rules, avatar and challenge, running on your device.

Here
Face modelGoogle's MediaPipe Face Landmarker (Apache-2.0 (Google)), pretrained and used as published; not retrained or re-measured here
Built hereFACS mood rules, relaxed-face calibration, smoothing, the avatar, Copy the mood and the slider mode
AccuracyNot measured: the rule weights were set by hand from the FACS definitions and checked on synthetic expressions, not on a labelled set of real faces
PrivacyVideo is processed in your browser and never uploaded or recorded; the challenge recap shows the avatar, not a photo
LimitsWorks best facing the camera in good light; side views, strong shadows, hands on the face or heavy glasses weaken the readings

Build notes

How it's built, step by step.

The real code behind this demo, taken from the repository when the site is built. Copy any piece.

In your browser · TypeScript

  1. 1

    Load Google's face model

    MediaPipe's Face Landmarker in video mode, asked for blendshapes and the head's transformation matrix; GPU first, CPU if the GPU stalls.

    src/lib/mediapipe.ts:36 · 13 lines

    /** Google's Face Landmarker in video mode, self-hosted; blendshapes and head pose only when a demo needs them. */
    export async function loadFaceLandmarker(extras = true): Promise<FaceLandmarker> {
      const [{ FaceLandmarker: FL }, vision] = await Promise.all([import("@mediapipe/tasks-vision"), visionFileset()]);
      return withGpuFallback((delegate) =>
        FL.createFromOptions(vision, {
          baseOptions: { modelAssetPath: "/mediapipe/face_landmarker.task", delegate },
          runningMode: "VIDEO",
          numFaces: 1,
          outputFaceBlendshapes: extras,
          outputFacialTransformationMatrixes: extras,
        }),
      );
    }
  2. 2

    Weigh the action units of each mood

    Subtracts the relaxed face, scales each blendshape group and takes a weighted mean per expression, keeping every unit's share for the 'Why' list.

    src/lib/face-mood.ts:39 · 13 lines

    /** Evidence (0–1) for each expression and how much each action unit contributed. */
    export function evidence(s: Shapes, baseline: Shapes = {}) {
      const at = (n: string) => Math.max(0, (s[n] ?? 0) - (baseline[n] ?? 0));
      const mean = (ns: string[]) => ns.reduce((a, n) => a + at(n), 0) / ns.length;
      const out = {} as Record<Exclude<Mood, "neutral">, { e: number; parts: { au: string; label: string; v: number; share: number }[] }>;
      for (const [mood, parts] of Object.entries(RULES) as [Exclude<Mood, "neutral">, Part[]][]) {
        const wsum = parts.reduce((a, p) => a + p.w, 0);
        const vals = parts.map((p) => clamp01((mean(p.shapes) - (p.minus ? mean(p.minus) : 0)) * p.gain));
        const e = parts.reduce((a, p, i) => a + p.w * vals[i], 0) / wsum;
        out[mood] = { e, parts: parts.map((p, i) => ({ au: p.au, label: p.label, v: vals[i], share: (p.w * vals[i]) / wsum })) };
      }
      return out;
    }
  3. 3

    Turn evidence into five probabilities

    Neutral gets the leftover when no recipe is strong; a sharpened softmax gives the bars and the top mood.

    src/lib/face-mood.ts:53 · 11 lines

    /** Probabilities over the five moods. Neutral wins when no expression has much evidence. */
    export function moodProbs(s: Shapes, baseline: Shapes = {}) {
      const ev = evidence(s, baseline);
      const strongest = Math.max(...Object.values(ev).map((x) => x.e));
      const score: Record<Mood, number> = { happy: ev.happy.e, surprised: ev.surprised.e, sad: ev.sad.e, angry: ev.angry.e, neutral: clamp01(0.45 - strongest) };
      const ex = MOODS.map((m) => Math.exp(SHARP * score[m]));
      const sum = ex.reduce((a, v) => a + v, 0);
      const probs = Object.fromEntries(MOODS.map((m, i) => [m, ex[i] / sum])) as Record<Mood, number>;
      const top = MOODS.reduce((a, m) => (probs[m] > probs[a] ? m : a), "neutral" as Mood);
      return { probs, top, ev };
    }
  4. 4

    Read the head's turn, nod and tilt

    Euler angles from MediaPipe's 4×4 transformation matrix, used to rotate and shift the avatar's features.

    src/lib/face-mood.ts:80 · 6 lines

    /** Head yaw / pitch / roll in degrees from MediaPipe's 4×4 facial transformation matrix (column-major). */
    export function headPose(m: ArrayLike<number> | undefined) {
      if (!m || m.length < 16) return { yaw: 0, pitch: 0, roll: 0 };
      const deg = 180 / Math.PI;
      return { yaw: Math.asin(Math.max(-1, Math.min(1, -m[2]))) * deg, pitch: Math.atan2(m[6], m[10]) * deg, roll: Math.atan2(m[1], m[0]) * deg };
    }
  5. 5

    Draw the mirror avatar

    An SVG face whose eyes, brows, cheeks and mouth follow the blendshapes, mirrored like a selfie.

    src/components/demos/face-mood-demo.tsx:48 · 61 lines

    /**
     * An original cartoon face driven by the blendshapes, mirrored like a selfie.
     * MediaPipe names sides from the camera's point of view, so "Left" shapes move the avatar's right side on screen.
     */
    export function Avatar({ s, pose, size = 220 }: { s: Shapes; pose: Pose; size?: number }) {
      const g = (n: string) => s[n] ?? 0;
      const dx = Math.max(-16, Math.min(16, -pose.yaw * 0.8));
      const dy = Math.max(-10, Math.min(10, pose.pitch * 0.5));
      const roll = Math.max(-25, Math.min(25, -pose.roll));
      const smile = (g("mouthSmileLeft") + g("mouthSmileRight")) / 2;
      const frown = (g("mouthFrownLeft") + g("mouthFrownRight")) / 2;
      const jaw = g("jawOpen");
      const pucker = g("mouthPucker");
      const browUp = g("browInnerUp");
      const eye = (side: "L" | "R") => {
        const k = side === "L" ? "Right" : "Left"; // screen-left eye follows MediaPipe's "Right" shapes

Notes

The decisions behind it.

Approach

  1. 1

    Ran Google's Face Landmarker in the browser with MediaPipe, self-hosted, GPU first with a CPU fallback, asking for blendshapes and the head's pose.

  2. 2

    Wrote the mood rules from FACS: happy (AU12, AU6), surprised (AU1, AU2, AU5, AU26), sad (AU15, AU1 without AU2, AU17) and angry (AU4, AU9, AU24, AU7), each a weighted mean of its action units.

  3. 3

    Subtracted the visitor's relaxed face, averaged over 1.5 seconds, and smoothed the blendshapes between frames so readings don't flicker.

  4. 4

    Drew an original avatar driven by the same blendshapes and head pose, built the Copy the mood challenge on top, and added sliders for visitors without a camera.

Honest note

The face model is Google's, used as published. The mood rules are mine and have not been measured on a labelled set of faces: the weights were set by hand from the FACS definitions and checked on synthetic expressions. Above all it reads expressions, not feelings: a smile can be polite, a frown can be concentration, and expressions vary between people and cultures, so it should never be used to judge anyone.

Reproduce: src/lib/face-mood.ts · src/components/demos/face-mood-demo.tsx

dataset source

Next project

Drowsiness Detector