←BaldanAll skillsLet's talk
Deep Learning & Vision lab

Computer vision

Boxes and labels for 80 everyday objects, live from your camera, computed on your own device.

80

Objects it knows

19,206

Candidate boxes per frame

7.3 MB

Model download

28 ms

Phone GPU (Google benchmark)

Overview

Object detection answers two questions at once: what is in the picture, and where. My original F1 project did it on racing footage; this rebuild turns it into a live tool anyone can point at their own surroundings.

It runs Google's EfficientDet-Lite0, trained on COCO, through MediaPipe in the browser. Every frame the network scores 19,206 candidate boxes; the best box per object survives, and a confidence slider and object filters decide what you see.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Frame

    camera or photo

    Processed on the device

  2. 02

    Resize

    320×320

    EfficientDet-Lite0 input

  3. 03

    Score

    19,206 boxes

    Candidate boxes per frame

  4. 04

    Merge

    best box

    Overlaps for one object merged

  5. 05

    Filter

    your threshold

    Confidence slider + object groups

  6. 06

    Show

    boxes + counts

    Live, with fps on your device

Inside the model

How a detector finds things.

Google's EfficientDet-Lite0 runs in your browser; here is what happens inside one pass, drawn on an illustrated desk.

RESIZED TO 320 × 320 PIXELSTIME PER PICTURE ON A PIXEL 6 PHONE CPU (GOOGLE'S BENCHMARK, MS)EfficientDet-Lite0 · int8 · 4.6 MB29.31EfficientDet-Lite0 · float16 · 7.25 MB53.97EfficientDet-Lite0 · float32 · 13.84 MB61.3EfficientDet-Lite2 · float16 · 12.14 MB198.77SSD MobileNetV2 · float16 · 5.92 MB37.35

The model and its data

80 everyday objects, 1.5 million labelled examples.

The detector is EfficientDet-Lite0, trained by Google on COCO and run in your browser by MediaPipe. It finds objects and draws a box around each one, frame after frame.

01

Frame

A camera frame or your photo, kept on the device

02

320×320

Resized for EfficientDet-Lite0

03

19,206 boxes

Every frame, the network scores this many candidate boxes

04

Best box wins

Overlapping boxes for the same object are merged (non-max suppression)

05

Your filter

Confidence slider and object groups, applied live

COCO: Common Objects in Context

The model was trained on COCO, a dataset of everyday scenes with 1.5 million labelled object instances across 80 categories. COCO's annotations are CC BY 4.0; its images come from Flickr under their own licences. The trained model is Google's, published for MediaPipe; it is used here as it is, not retrained.

80

object types

1.5 million

labelled objects

320×320

model input

7.3 MB

model download

COCO dataset

Column dictionary

● used by the model

ColumnTypeWhat it means
imagephotoAny camera frame, resized to 320×320 for the network
boxx · y · w · hWhere the object is, in pixels of the original frame
category1 of 80Which COCO object it is: person, cup, laptop, car…
score0–1How sure the model is; your slider decides the cut-off

Try it

Point your camera at anything.

Your desk, a cup, your phone, the street outside, or yourself. Boxes, labels and counts update live; on a phone you can switch to the back camera.

Live demo

runs in your browser

Turn on your camera and point it at your desk, a cup, your phone or yourself.

Video and photos are processed on your device; nothing is uploaded or recorded.

0

objects in view

0

people

Only show

In this frame

Start the camera or pick a photo.

What it can see

Eighty objects, nothing more.

The 80 COCO objects, grouped as in the demo's filter

Anything outside this list is invisible to the model, or gets the name of the closest thing it knows. There is a 'car' but no 'racing car', a 'cup' but no 'mug'.

People1

person

Vehicles8

bicycle · car · motorcycle · airplane · bus · train · truck · boat

Electronics6

tv · laptop · mouse · remote · keyboard · cell phone

Food & kitchen17

bottle · wine glass · cup · fork · knife · spoon · bowl · banana · apple · sandwich · orange · broccoli · carrot · hot dog · pizza · donut · cake

Home18

chair · couch · potted plant · bed · dining table · toilet · microwave · oven · toaster · sink · refrigerator · book · clock · vase · scissors · teddy bear · hair drier · toothbrush

Animals10

bird · cat · dog · horse · sheep · cow · elephant · bear · zebra · giraffe

Bags & sport15

backpack · umbrella · handbag · tie · suitcase · frisbee · skis · snowboard · sports ball · kite · baseball bat · baseball glove · skateboard · surfboard · tennis racket

Street5

traffic light · fire hydrant · stop sign · parking meter · bench

Why this model

7.3 MB, about 28 ms a frame on a phone GPU.

Google publishes five ready-made detectors for MediaPipe. EfficientDet-Lite0 in float16 is the balance: small enough to download on a phone, fast on the GPU, and the variant Google recommends. Your own speed shows live in the demo's corner.

Speed per frame (Google's benchmark, Pixel 6)

On the GPU the Lite0 variants and SSD MobileNetV2 all run near 28 ms, fast enough for live video. Lite2 sees more detail (448 px input) but costs far more, especially on the CPU. The int8 version has no GPU figure.

CPUGPU
–25050029–Lite0 int85428Lite0 float166128Lite0 float3219947Lite2 float163728SSD float16
Download size

The browser downloads the model once. float16 halves the float32 file at no GPU speed cost; int8 is smaller still but CPU-only in this table.

0.010.020.04.6Lite0 int8Lite0 float1613.8Lite0 float32Lite2 float16SSD float16
What this project is, and isn't

A rebuild of my F1 object detection project as a live, private, in-browser tool.

Here
The modelGoogle's EfficientDet-Lite0, pretrained on COCO, used as published
Built hereCamera pipeline, coordinate mapping, mirroring, confidence filter, object groups, live counts, photo mode
Runs onYour device: WebAssembly and your GPU, self-hosted files, no server
AccuracyNot re-measured here; Google's page publishes speeds, not accuracy, for these models
LimitsOnly the 80 COCO objects; small, distant, blurry or partly hidden objects are often missed

Build notes

How it's built, step by step.

The real code behind this demo, taken from the repository when the site is built. Copy any piece.

In your browser · TypeScript

  1. 1

    Start MediaPipe on the GPU, fall back to the CPU

    Tries the GPU delegate with a timeout and quietly retries on the CPU, so the demo starts on any device.

    src/lib/mediapipe.ts:17 · 9 lines

    export async function withGpuFallback<T>(make: (delegate: "GPU" | "CPU") => Promise<T>, timeoutMs = 8000): Promise<T> {
      // some browsers stall on GPU setup (blocked WebGL, background tabs): give it a few seconds, then use the CPU
      const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("GPU setup timed out")), timeoutMs));
      try {
        return await Promise.race([make("GPU"), timeout]);
      } catch {
        return await make("CPU");
      }
    }
  2. 2

    Turn detections into labelled boxes

    Keeps the best category per box with its score, ready for the confidence slider and object filters.

    src/components/demos/object-detection-demo.tsx:50 · 9 lines

    function toFound(dets: Detection[]): Found[] {
      return dets
        .map((d) => {
          const c = d.categories?.[0];
          const b = d.boundingBox;
          return c && b ? { name: c.categoryName, score: c.score, box: { x: b.originX, y: b.originY, w: b.width, h: b.height } } : null;
        })
        .filter((f): f is Found => !!f && f.name !== "???");
    }
  3. 3

    Draw boxes on a mirrored camera view

    Mirrors the box coordinates for the selfie view but not the labels, so the text stays readable.

    src/components/demos/object-detection-demo.tsx:60 · 27 lines

    /** Draw boxes and labels. When the video is shown mirrored, boxes are mirrored by coordinates so the text stays readable. */
    function drawBoxes(canvas: HTMLCanvasElement, w: number, h: number, found: Found[], mirror: boolean) {
      const ctx = canvas.getContext("2d");
      if (!ctx) return;
      if (canvas.width !== w || canvas.height !== h) {
        canvas.width = w;
        canvas.height = h;
      }
      ctx.clearRect(0, 0, w, h);
      const s = Math.max(w, h) / 640;
      ctx.font = `600 ${Math.round(13 * s)}px var(--font-geist-sans), system-ui, sans-serif`;
      for (const f of found) {
        const color = GROUP_COLOR[GROUP_OF[f.name]] ?? "#0b0b0b";
        const x = mirror ? w - f.box.x - f.box.w : f.box.x;
        ctx.lineWidth = 3 * s;
        ctx.strokeStyle = color;

Notes

The decisions behind it.

Approach

  1. 1

    Compared the five detectors Google publishes for MediaPipe on its Pixel 6 benchmark and their file sizes; picked EfficientDet-Lite0 in float16.

  2. 2

    Ran it in the browser with MediaPipe, self-hosted, GPU first with a CPU fallback; the library loads only when you start the demo.

  3. 3

    Mapped boxes onto a mirrored selfie view without mirroring the labels, and added a front/back camera switch for phones.

  4. 4

    Kept the model's own threshold low and filtered live in the page, so the confidence slider and object groups react instantly.

Google's Pixel 6 benchmark: GPU ms per frame · download size

EfficientDet-Lite0 · int8no GPU figure · 4.6 MB
EfficientDet-Lite0 · float1627.97 ms · 7.3 MBused
EfficientDet-Lite0 · float3227.83 ms · 13.8 MB
EfficientDet-Lite2 · float1647.31 ms · 12.1 MB
SSD MobileNetV2 · float1628.16 ms · 5.9 MB

Honest note

The detector is Google's, pretrained on COCO and used as published; I built the tool around it and did not retrain or re-evaluate it, and Google's page gives speeds but no accuracy figures for these models. It only knows COCO's 80 objects, so a racing car is just a car, and small, distant, blurry or partly hidden objects are often missed. Lowering the confidence slider finds more of them, along with more wrong boxes.

Reproduce: src/components/demos/object-detection-demo.tsx

dataset source

Next project

Live Forecast Lab