FImageSearch: Multimodal Search for an AI Stylist Agent
Case StudyA client was building a personal AI stylist that lives in messengers: you send it a street-style photo, it finds every piece of the look in a shop catalog. The stylist agent needs a search core it can call — and that core is what this project is: a shop-the-look engine that detects every garment on a photo and matches each one against 105,100 catalog items in a couple hundred milliseconds. Photo, text, photo + text refinement, manual crop — all four query modes land in the same vector space. The first working demo went live in one evening.
Recorded on a phone against the live demo: full-look analysis, match percentages, manual crop and text refinement over the real 105k catalog.
The demo runs on an on-demand GPU box, so it sleeps between showings — if the link is down, get in touch and I will wake it up for you.
The numbers
How a photo becomes ranked results
One HTTP request runs the whole pipeline. The trick is batching: every garment crop plus the person crop are embedded in a single GPU forward pass, and all slots go to the vector DB as one batched query.
photo ──▶ person detection RT-DETRv2, ~15 ms
──▶ garments in person crop YOLOS-Fashionpedia, 46 classes
──▶ 9 slots, head to toe headwear → … → shoes, max 7 per look
──▶ ONE GPU batch all crops + person, fashionSigLIP, 768-d
──▶ ONE batched Qdrant query hard filters: group / category / gender
──▶ CIELAB color re-rank ──▶ ranked results with % matchVector similarity alone is not enough — the search is hybrid. Hard metadata filters ride on top of every query: a shoe crop only searches Shoes, a men’s look only searches the men’s sections, and jacket families are excluded from top slots so a layered outfit doesn’t drown in puffer jackets. Inside those fences, the vector ranks by similarity.
What the models couldn’t do out of the box
The interesting engineering is in the gap between “embeddings work” and “results look right”. Everything below was found on real photos and fixed with measurements, not vibes:
SigLIP is nearly color-blind on basics
Black vs beige basic tees differ by Δcos ≈ 0.01 — even on a clean fabric crop. Fix: pre-computed dominant CIELAB color for all 105k items and a color re-rank (weight 0.12, lightness at half weight, since user photos live in shadows and catalogs in studio light).
The catalog skews the gender vote
First gender-detection attempt — majority vote over search results — deadlocked: the catalog is 3:1 womenswear, so male crops legitimately pull female items. Fix: a zero-shot probe comparing the person crop against “a man” / “a woman” text anchors in the same batch. ~2 ms, 7/7 on reference photos.
Phones read as sunglasses
In mirror selfies the fashion detector sees the phone at face level as glasses (it returned phone cases for a glasses slot). Fix: cell-phone boxes from the person detector’s same forward pass — zero extra cost — suppress glasses/headwear that overlap them.
Padding pulls the jacket into the shirt
A top worn under an open jacket embeds badly with context padding — the crop drags the blazer in and returns jackets. Verified by sweep: layered tops are embedded with no padding at all, at a lowered acceptance floor.
A pair of shoes is mostly asphalt
The union box around both shoes mid-stride is dominated by road surface, which poisons the embedding. The UI highlights both shoes, but only the larger, less occluded one is embedded.
Raw cosine scores make no visual sense
Text and image cosines live in different ranges; a linear clamp showed every result as 100%. Fix: per-modality logistic calibration, so the “97% match” badge actually ranks results and never quite reaches 100.
What I built
Two-stage detection
RT-DETRv2 finds the people, picks the main person; YOLOS-Fashionpedia (46 garment classes) runs inside that crop — 2–4× effective resolution on small items like glasses and shoes.
Nine-slot look taxonomy
Detections resolve into head-to-toe slots (headwear → shoes) with rules for dress-vs-top+bottom conflicts, layered outfits and slot-specific crop padding, calibrated by sweeps on reference photos.
One shared vector space
marqo-fashionSigLIP embeds images and text into the same 768-d cosine space. Photo + “but in red” is a weighted vector sum (0.6 image + 0.4 text) — and multilingual queries work for free.
Hybrid search over Qdrant
Batched queries with hard payload filters per slot (group, category, gender section) plus keyword indexes — vector similarity ranks only within the right shelf of the shop.
Resumable indexing that survived a dead server
Embeddings persist as incremental shards; the index restores without re-embedding. The original GPU box was physically destroyed mid-project, then the whole thing migrated Germany → Poland → California — no GPU re-run needed.
Honest-UX frontend
Next.js 14 mobile-first UI: auto-analysis on upload, tap-synced dots and result rows, WB-style manual cropper, % on every card and a “no exact match — closest options” banner instead of pretending.
Stack
Python · FastAPI · PyTorch fp16 · open_clip · marqo-fashionSigLIP · RT-DETRv2 · YOLOS-Fashionpedia · Qdrant · Next.js 14 · Tailwind + shadcn/ui · Vast.ai GPU · Cloudflare Tunnel + R2
Want visual search over your catalog?
The core is domain-agnostic: detector → slots → embeddings → filtered vector search → re-rank. Fashion today; furniture, interiors or any visual catalog with a week of calibration. If your product needs “find me this” — let’s talk.
Get in touchSolo project, July 2026. Demo over the public H&M Kaggle dataset (2022); prices in the demo are illustrative. Client and commercial details are under NDA.