Computer vision
Boxes and labels for 80 everyday objects, live from your camera, computed on your own device.
80
Objects it knows
19,206
Candidate boxes per frame
7.3 MB
Model download
28 ms
Phone GPU (Google benchmark)
Overview
Object detection answers two questions at once: what is in the picture, and where. My original F1 project did it on racing footage; this rebuild turns it into a live tool anyone can point at their own surroundings.
It runs Google's EfficientDet-Lite0, trained on COCO, through MediaPipe in the browser. Every frame the network scores 19,206 candidate boxes; the best box per object survives, and a confidence slider and object filters decide what you see.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Frame
camera or photo
Processed on the device
- 02
Resize
320×320
EfficientDet-Lite0 input
- 03
Score
19,206 boxes
Candidate boxes per frame
- 04
Merge
best box
Overlaps for one object merged
- 05
Filter
your threshold
Confidence slider + object groups
- 06
Show
boxes + counts
Live, with fps on your device
Inside the model
How a detector finds things.
Google's EfficientDet-Lite0 runs in your browser; here is what happens inside one pass, drawn on an illustrated desk.
The model and its data
80 everyday objects, 1.5 million labelled examples.
The detector is EfficientDet-Lite0, trained by Google on COCO and run in your browser by MediaPipe. It finds objects and draws a box around each one, frame after frame.
01
Frame
A camera frame or your photo, kept on the device
02
320×320
Resized for EfficientDet-Lite0
03
19,206 boxes
Every frame, the network scores this many candidate boxes
04
Best box wins
Overlapping boxes for the same object are merged (non-max suppression)
05
Your filter
Confidence slider and object groups, applied live
COCO: Common Objects in Context
The model was trained on COCO, a dataset of everyday scenes with 1.5 million labelled object instances across 80 categories. COCO's annotations are CC BY 4.0; its images come from Flickr under their own licences. The trained model is Google's, published for MediaPipe; it is used here as it is, not retrained.
80
object types
1.5 million
labelled objects
320×320
model input
7.3 MB
model download
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| image | photo | Any camera frame, resized to 320×320 for the network |
| box | x · y · w · h | Where the object is, in pixels of the original frame |
| category | 1 of 80 | Which COCO object it is: person, cup, laptop, car… |
| score | 0–1 | How sure the model is; your slider decides the cut-off |
Try it
Point your camera at anything.
Your desk, a cup, your phone, the street outside, or yourself. Boxes, labels and counts update live; on a phone you can switch to the back camera.
Live demo
runs in your browser
Turn on your camera and point it at your desk, a cup, your phone or yourself.
Video and photos are processed on your device; nothing is uploaded or recorded.
0
objects in view
0
people
Only show
In this frame
Start the camera or pick a photo.
What it can see
Eighty objects, nothing more.
Anything outside this list is invisible to the model, or gets the name of the closest thing it knows. There is a 'car' but no 'racing car', a 'cup' but no 'mug'.
People1
person
Vehicles8
bicycle · car · motorcycle · airplane · bus · train · truck · boat
Electronics6
tv · laptop · mouse · remote · keyboard · cell phone
Food & kitchen17
bottle · wine glass · cup · fork · knife · spoon · bowl · banana · apple · sandwich · orange · broccoli · carrot · hot dog · pizza · donut · cake
Home18
chair · couch · potted plant · bed · dining table · toilet · microwave · oven · toaster · sink · refrigerator · book · clock · vase · scissors · teddy bear · hair drier · toothbrush
Animals10
bird · cat · dog · horse · sheep · cow · elephant · bear · zebra · giraffe
Bags & sport15
backpack · umbrella · handbag · tie · suitcase · frisbee · skis · snowboard · sports ball · kite · baseball bat · baseball glove · skateboard · surfboard · tennis racket
Street5
traffic light · fire hydrant · stop sign · parking meter · bench
Why this model
7.3 MB, about 28 ms a frame on a phone GPU.
Google publishes five ready-made detectors for MediaPipe. EfficientDet-Lite0 in float16 is the balance: small enough to download on a phone, fast on the GPU, and the variant Google recommends. Your own speed shows live in the demo's corner.
On the GPU the Lite0 variants and SSD MobileNetV2 all run near 28 ms, fast enough for live video. Lite2 sees more detail (448 px input) but costs far more, especially on the CPU. The int8 version has no GPU figure.
The browser downloads the model once. float16 halves the float32 file at no GPU speed cost; int8 is smaller still but CPU-only in this table.
A rebuild of my F1 object detection project as a live, private, in-browser tool.
| Here | |
|---|---|
| The model | Google's EfficientDet-Lite0, pretrained on COCO, used as published |
| Built here | Camera pipeline, coordinate mapping, mirroring, confidence filter, object groups, live counts, photo mode |
| Runs on | Your device: WebAssembly and your GPU, self-hosted files, no server |
| Accuracy | Not re-measured here; Google's page publishes speeds, not accuracy, for these models |
| Limits | Only the 80 COCO objects; small, distant, blurry or partly hidden objects are often missed |
Build notes
How it's built, step by step.
The real code behind this demo, taken from the repository when the site is built. Copy any piece.
In your browser · TypeScript
- 1
Start MediaPipe on the GPU, fall back to the CPU
Tries the GPU delegate with a timeout and quietly retries on the CPU, so the demo starts on any device.
src/lib/mediapipe.ts:17 · 9 lines
export async function withGpuFallback<T>(make: (delegate: "GPU" | "CPU") => Promise<T>, timeoutMs = 8000): Promise<T> { // some browsers stall on GPU setup (blocked WebGL, background tabs): give it a few seconds, then use the CPU const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("GPU setup timed out")), timeoutMs)); try { return await Promise.race([make("GPU"), timeout]); } catch { return await make("CPU"); } } - 2
Turn detections into labelled boxes
Keeps the best category per box with its score, ready for the confidence slider and object filters.
src/components/demos/object-detection-demo.tsx:50 · 9 lines
function toFound(dets: Detection[]): Found[] { return dets .map((d) => { const c = d.categories?.[0]; const b = d.boundingBox; return c && b ? { name: c.categoryName, score: c.score, box: { x: b.originX, y: b.originY, w: b.width, h: b.height } } : null; }) .filter((f): f is Found => !!f && f.name !== "???"); } - 3
Draw boxes on a mirrored camera view
Mirrors the box coordinates for the selfie view but not the labels, so the text stays readable.
src/components/demos/object-detection-demo.tsx:60 · 27 lines
/** Draw boxes and labels. When the video is shown mirrored, boxes are mirrored by coordinates so the text stays readable. */ function drawBoxes(canvas: HTMLCanvasElement, w: number, h: number, found: Found[], mirror: boolean) { const ctx = canvas.getContext("2d"); if (!ctx) return; if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h; } ctx.clearRect(0, 0, w, h); const s = Math.max(w, h) / 640; ctx.font = `600 ${Math.round(13 * s)}px var(--font-geist-sans), system-ui, sans-serif`; for (const f of found) { const color = GROUP_COLOR[GROUP_OF[f.name]] ?? "#0b0b0b"; const x = mirror ? w - f.box.x - f.box.w : f.box.x; ctx.lineWidth = 3 * s; ctx.strokeStyle = color;
Notes
The decisions behind it.
Approach
- 1
Compared the five detectors Google publishes for MediaPipe on its Pixel 6 benchmark and their file sizes; picked EfficientDet-Lite0 in float16.
- 2
Ran it in the browser with MediaPipe, self-hosted, GPU first with a CPU fallback; the library loads only when you start the demo.
- 3
Mapped boxes onto a mirrored selfie view without mirroring the labels, and added a front/back camera switch for phones.
- 4
Kept the model's own threshold low and filtered live in the page, so the confidence slider and object groups react instantly.
Google's Pixel 6 benchmark: GPU ms per frame · download size
Honest note
The detector is Google's, pretrained on COCO and used as published; I built the tool around it and did not retrain or re-evaluate it, and Google's page gives speeds but no accuracy figures for these models. It only knows COCO's 80 objects, so a racing car is just a car, and small, distant, blurry or partly hidden objects are often missed. Lowering the confidence slider finds more of them, along with more wrong boxes.
Reproduce: src/components/demos/object-detection-demo.tsx
dataset sourceNext project
Live Forecast Lab