Point the camera at a broken part, find it on the shelf
A cross-platform app that identifies a plumbing or electrical part from a photo and points to the aisle it's in, running the recognition on the phone itself, because the stores have almost no signal past the front doors.
How the work was scoped
E-commerce & Retail
14 weeks
Time & materials
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
Recognition on the phone
Identification runs on-device, so the store's dead coverage stops mattering.
A quantized TensorFlow Lite model runs on the phone, so identification takes about 200ms and works in the aisle where there's no signal at all, which is the place the app is actually used. Inference runs on the device's own accelerator where one exists and falls back to CPU where it doesn't, and the model file ships with the binary, so there's no download on first run.
- Quantised TFLite model, ~200ms on-device inference
- Works in aisles with no signal, which is where it's used
- Model ships in the binary, no first-run download
What actually lives on the phone: a 34 MB quantized model shipped inside the app, the last scan named in 208 ms on the neural engine with a CPU fallback, the eight families making up its 400 fittings, and six store maps held locally.
The handoff in place of a confident wrong answer: four candidates all under the 0.45 threshold, an offer to search in case the part is outside the 400, and a counter ticket carrying the photograph.
Scoped to what returns
The model covers the 400 fittings behind most returns, not an open catalog.
The model recognizes the four hundred fittings behind most returns, not the whole catalog. A narrow model is small enough to run on a mid-range phone and accurate enough to be trusted, where an open-catalog model would have been neither. Anything outside that set returns not recognized and offers search, which is a better answer than a confident wrong part number.
- Scoped to the 400 fittings behind most returns
- Small enough for a mid-range phone, accurate enough to trust
- Out-of-set inputs decline instead of guessing
A cached aisle map
The cached floor plan walking the customer to aisle seven, bay four, stamped with how old the map is, with every bay on that run and what sits in it.
Each store's map held locally and refreshed opportunistically when signal is available.
Once a part is named it still has to be found, so each store's aisle map and stock snapshot are cached on the device and refreshed opportunistically whenever there's signal: at the counter, in the parking lot, anywhere. With a connection the app resolves against live stock; without one it shows the cached position and says how old it is, never pretending to be current.
- Per-store aisle map and stock cached locally
- Refreshed opportunistically whenever signal appears
- Cached results state their age instead of implying freshness
What we were brought in to do
Customers walk in holding a broken fitting and no idea what it's called. We built an iOS and Android app that names the part from a photo and shows where it sits in that specific store, and made the whole recognition path run on-device, because a steel-shelved hardware store is a dead zone.
A hardware retailer where the single biggest consumer of counter time was returns, and most returns were the same failure: a customer took home a fitting that looked right. They arrive holding a broken part and no vocabulary for it, and naming it correctly requires a staff member who knows fittings, of whom there are two per store on a good day.
Mobile Engineering & AI
Where the old way broke
Naming a part was the whole bottleneck. Customers waited for a staff member who knew fittings, guessed, and took the wrong thing home, and returns were the single biggest source of counter time. A cloud-based scanner was tried first and failed for a simple reason: there's no usable signal in the aisles where people actually stand.
A cloud scanner had already been tried and abandoned. The reason was mundane and decisive: a hardware store is steel shelving in a metal shed, and there's no usable signal in the aisles where customers actually stand holding the part. The app worked in the parking lot and at the counter, which are the two places nobody needs it.
We shipped a single React Native codebase with a compact recognition model running on the phone, so identification works with the network completely off. Once a part is named, the app resolves it against that store's live stock and aisle map when signal returns at the counter, and falls back to a cached map when it doesn't.
What we built together
Ran recognition fully on-device so the aisles' dead zone stopped mattering
On-device was settled by the failed cloud attempt: the aisles have no signal, and that's where the customer and the broken part actually are.
Scoped the model to the 400 fittings that drove most of the returns
The model covers the four hundred fittings behind most returns, not the whole catalog, because a narrow model is both small enough and accurate enough.
Cached each store's aisle map locally, refreshed whenever the app had signal
Each store's aisle map and stock snapshot cache locally and refresh opportunistically, so a named part still has a location when there's no connection.
Handed off to a staff lookup when confidence was low, with no guessing
Below the confidence threshold the app declines and offers a staff lookup. It never returns its best guess.
Phase by phase
Phase 1: Decide where it runs
On-device, because of the aisles
Established that recognition had to run fully on-device, since the aisles have almost no signal.
- Feasibility study
- Model size budget
Phase 2: Scope the model
400 fittings, not everything
Scoped the recognition model to the 400 fittings that drove most returns, trading breadth for accuracy.
- Training set
- Accuracy benchmarks
Phase 3: Cache locally
The map, held on the phone
Cached each store's aisle map locally and refreshed it whenever the app had connectivity.
- Local cache
- Sync strategy
Phase 4: Handle doubt
Hand off, do not guess
Built a staff-lookup handoff for low-confidence matches instead of returning a confident wrong answer.
- Confidence thresholds
- Handoff flow
- Store release
A part named but out of stock here, with the four other stores that have it from the cached stock snapshot, its age stated, and what's usually bought alongside.
Operational results after launch
-46%
Wrong-part returns
Works
Identification with signal off
Under 4s
Median time to name a part
Wrong-part returns compares the two quarters after launch with the two before, at stores where the app was promoted. Identification with signal off is a capability claim, not a measurement. Median time to name a part is measured in-app from camera open to a named result, excluding the low-confidence handoffs.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
About our collaboration
- 01
A cross-functional team of 5 worked on a time & materials basis over 14 weeks, covering React Native build, On-device vision, Store release. We ran daily standups with their own lead in the room, and a demo at the end of every sprint. Scope changed twice during the engagement, and both times the change was priced and agreed before work started.
Time and materials, because the model's scope genuinely couldn't be fixed in advance: how narrow it had to be to run acceptably on a mid-range phone was something to be discovered. Recognition accuracy was measured against real returned parts photographed in real aisle lighting, not a clean dataset, and that's what set the four-hundred-item boundary.
What we'd carry into the next one
- 01
Scoping the model down was what made on-device accuracy possible. Breadth would have cost both.
Breadth and on-device accuracy trade against each other directly: an open-catalogue model would have been too large to run and too uncertain to trust.
- 02
The signal dead zone was the defining constraint, and designing around it was cheaper than fighting it.
The dead zone wasn't an edge case to engineer around. It was the operating environment, and accepting it produced a simpler system than fighting it would have.
- 03
A confident wrong part is worse than no answer; the handoff was a feature, not a fallback.
Declining is the whole value proposition here. The failure being solved is a customer confidently taking the wrong part home, and a wrong answer reproduces it exactly.
One scan, in an aisle with no signal
Named, located and checked for stock without a single bar of signal.
Choose the part in the customer’s hand and follow it from the camera frame to the shelf. One is on the shelf, one is out of stock here, and one the model won’t name. Switch steps, or use the arrow keys once one is focused.
Named: 15mm compression elbow
Quantised TensorFlow Lite on the phone's accelerator (CPU where there isn't one), about 200 ms. The white mark is the 0.45 threshold; the top score clears it, so the part is named.
Illustrative: runner-up scores and the 600 ms “matching” pause are for reading, not measurements; on-device inference is about 200 ms. Parts, bays, stock counts and store names are invented.
From a photo in a dead zone to the right bay, with the network optional
One React Native codebase on iOS and Android. The dead zone is the operating environment, not an edge case, so every step up to the bay runs on the handset and the network only ever makes an answer fresher.
- 01 · TriggerCamera frame in the aisleCaptured where the customer stands with the broken part, which is where there's no signal. Nothing waits on a network.
- 02 · InferenceQuantised TFLite on the phoneAbout 200 ms on the device's accelerator, falling back to CPU. The model ships in the binary: no first-run download.
- 03 · Decision400 fittings, one thresholdScoped to the fittings behind most returns. Below the confidence threshold, or outside the set, it declines and offers search or a staff lookup.
- 04 · StatePer-store map + stock in SQLiteEach store's aisle map and stock snapshot cached on the phone, so a named part still has a location with no connection.
- 05 · DeliveryLive stock when signal returnsRefreshed opportunistically: counter, parking lot, anywhere. With signal it resolves live; without, the cached answer states its age.
No signal, no guessing, no stale surprises
Keeping the wrong part out of the customer’s hand
Works where the part is
Recognition runs entirely on the phone, so identification works with the signal completely off: in the steel-shelved aisle, well beyond the parking lot and the counter where a cloud scanner used to work.
Declines rather than guesses
Below the confidence threshold, or for anything outside the fittings it covers, the app returns no part and offers search or a staff lookup. A confident wrong answer is the failure it was built to stop.
Cached answers say how old they are
Aisle positions and stock come from the per-store copy on the phone when there's no connection, and show its age. With signal the app resolves against live stock, so the copy never poses as current.
Need an app that recognises things on the phone, where there’s no signal? Scope your build in 3 minutes.
Scope your buildNearby engagements
Data & AnalyticsA checkout that stopped losing sales to a 6-second load
Profiling found the real bottleneck behind a slow checkout (an N+1 query and an oversized bundle) and tuned both, with before-and-after metrics locked in as a baseline against future regressions.
E-commerce & Retail · 5 weeks
Web PlatformsA reader people finish, and a library that remembers where they stopped
A reading platform for a comics catalog: a browse surface people can actually navigate, a reader that gets out of the way, and a history that puts everyone back on the page they left.
E-commerce & Retail · 18 weeks
Web PlatformsOne number, a whole handset, and a database that keeps up with 125 brands
A metered IMEI lookup service that turns fifteen digits into a device, its specifications and its status. It's sold three ways to three audiences and backed by a device database that maintains itself.
Consumer Electronics · 22 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
