Keeping AI in production healthy, month after month
A managed retainer that monitors, tunes, and improves live AI systems, from support agents to forecasting, so they keep getting better after launch.
The brief, in specifics
E-commerce & Retail
Ongoing
Ongoing retainer, phased
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
What went wrong, and when
AI systems drift, prompts go stale, and edge cases pile up. Without ownership, quality quietly erodes after launch.
Every one of the four had the same shape of gap. Latency and error rates were monitored because that's what infrastructure monitoring does; response quality wasn't, because nobody had defined what quality meant for these systems. Drift was therefore invisible by construction: the systems kept answering, quickly and successfully, with answers that had quietly got worse.
We put monitoring, evaluation, and a tuning cadence around every live AI system, with a monthly review of quality, cost, and new edge cases to fix.
Phase by phase
Phase 1: Model Drift & Prompt Audit
Production Evaluation Baseline
Audited production AI performance, hallucination rates, and prompt degradation across live retail customer service agents.
- Model Drift Audit
- SLA Monitor Architecture
- Quality Baseline Report
Phase 2: Automated Eval & Guardrails
Regression Suite & Safety Filters
Engineered continuous evaluation pipelines testing every agent output against regression suites and safety guardrails.
- Eval Suite Pipeline
- Hallucination Detector
- Safety Filter Layer
Phase 3: Cost Optimization & Caching
Token Compression & Semantic Cache
Optimized prompt tokens and implemented semantic caching, reducing monthly API costs by 32%.
- Token Optimization Engine
- Semantic Cache Layer
- Cost Reduction Report
Phase 4: Ongoing SLA & Monthly Tuning
Continuous Improvement Cadence
Delivered monthly performance reports, prompt updates, and new feature integrations under guaranteed SLA response times.
- Monthly SLA Reports
- Continuous Improvement Log
- Prometheus Dashboard
The monthly report for the business: uptime per system, every incident on its own clock, how the retainer hours were spent, and what to fix, cap or retire.
The numbers, before and after
99.9%
Uptime
Pre-impact
Quality drift caught
−21%
Model cost
Uptime is per system against the agreed SLA, never aggregated. Quality drift caught before impact is a count of regressions the evaluation pipeline blocked or flagged before they reached customers, which is the claim this retainer exists to make. Cost is monthly inference spend, compared against the three months before the retainer began.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
The engagement
Several AI systems were live and needed watching and improving. We run a managed retainer that keeps everything healthy and getting better.
A retailer with four AI systems already in production, each built by a different team or vendor and none of them owned after launch. The engagement started from an incident: a model provider deprecated a version, the replacement behaved differently on the retailer's edge cases, and nobody noticed for eleven days because nothing was watching quality. Only uptime.
Managed Operations
How it was handled
- 01
Set up continuous evaluation pipelines for production prompts
The evaluation set had to be built from scratch, because none of the four live systems had a definition of what a good answer looked like.
- 02
Monitored response quality, latency, and cost per query daily
Daily review was set as a cadence, not a dashboard, because the four systems had all had monitoring and none of it had been looked at.
- 03
Ran prompt tuning sprints every two weeks to catch new edge cases
Tuning runs every two weeks against the edge cases those two weeks actually produced, so nothing waits for a complaint to get scheduled.
- 04
Reported business impact, SLA compliance, and recommendations monthly
Reporting monthly instead of continuously was deliberate: a dashboard nobody opens had already been tried on all four systems and had changed nothing.
Continuous LLM Evaluation Pipeline
Automated regression tests and hallucination detectors running against live production traffic.
A frozen evaluation set is replayed against every prompt or model change before it ships, and a sample of live traffic is scored continuously afterward. Hallucination checks are grounded wherever possible, never just model-graded: an assertion is verified against the source the system retrieved, so the detector isn't another model's opinion. A regression blocks the release outright; it doesn't just file a ticket.
- Frozen eval set replayed before every prompt or model change
- Live traffic sampled and scored continuously
- Regressions block the release outright
A prompt version scored overnight against 2,400 recorded conversations: pass rate by intent with the two regressions picked out in red, the grounded check that failed against its retrieved source, and the gate that held the release.
Token & Cost Optimizer
Where the month's token spend went, feature by feature, after the semantic cache and prompt compression took their cut: similarity thresholds per surface, stable instructions in a cached prefix, and the few-shot examples that were cut.
Semantic caching layer and prompt compression that lowers monthly inference bills.
A semantic cache sits in front of the model, so a question already answered this hour returns from store instead of being billed again, with the similarity threshold tuned per surface, because a support answer tolerates near-matching and a legal answer doesn't. Prompts were compressed by moving stable instructions into cached prefixes and cutting few-shot examples that measurement showed were doing nothing.
- Semantic cache with per-surface similarity thresholds
- Stable instructions moved into cached prefixes
- Few-shot examples cut where measurement showed no effect
24/7 Model SLA Monitoring
Prometheus & Grafana dashboards tracking latency, drift, and accuracy metrics in real time.
Latency, cost per request, retrieval hit rate and eval scores are exported to Prometheus as first-class metrics and alerted on with thresholds agreed as an SLA, never chosen by whoever built the dashboard. Drift is watched on the input distribution as well as the output, because the first sign of trouble is usually that people started asking a different kind of question.
- Latency, cost, retrieval hit rate and eval scores as SLA metrics
- Alert thresholds agreed contractually, never picked ad hoc
- Input drift watched alongside output quality
Answers against the three-second target hour by hour, the SLA-agreed metrics, the live stream of questions as they land in store and online, and drift watched on the question mix.
Working inside their operation
- 01
A cross-functional team of 6 worked on an ongoing retainer, covering Monitoring, Model tuning, Continuous improvement. We ran a standing mid-week checkpoint and written decisions in place of status meetings. Nothing shipped without a live demo first.
The cadence is the product: continuous evaluation, daily cost and latency review, a tuning sprint every two weeks, and a monthly report on quality, spend and edge cases. The monthly report goes to the business, not to engineering, because the decisions that matter (what to fix, what to cap, what to retire) are commercial ones.
What changed in the runbook
Continuous prompt tuning and evaluation maintained 99.8% production uptime.
Uptime was never the hard part. The retainer earns its place by catching the quality regressions that leave uptime untouched, which is exactly what the deprecation incident was.
Semantic caching and token optimization reduced monthly LLM API costs by 32%.
Most of the cost reduction came from the cache and from prompt prefixes, not from a cheaper model. The quality trade nobody wanted was never necessary.
Proactive model drift detection prevented quality decay as underlying models updated.
Watching the input distribution is what makes drift detectable early: people start asking a different kind of question before the answers visibly get worse.
The average said ship it. Two intents said no.
A candidate prompt for the support agent, replayed overnight against 2,400 recorded conversations: the run, the regressions the per-intent rule caught, and what the gate did about them. Switch tabs, or use the arrow keys once one is focused.
Every change passes the same set; every answer stays on a metric someone agreed to
The four systems all had infrastructure monitoring, and it watched uptime. This is the loop wrapped around them so quality is watched too, from the question as it lands to the decision the business makes each month.
- 01 · TrafficLive questions, store and onlineA sample of live traffic is scored continuously after every release, and the input distribution is watched as well as the answers.
- 02 · GateFrozen set · 2,400 conversationsReplayed against every prompt or model change before it ships. A regression blocks the release outright.
- 03 · ServeSemantic cache + cached prefixesA question already answered returns from store, with the similarity threshold tuned per surface; stable instructions sit in cached prefixes.
- 04 · WatchPrometheus + GrafanaLatency against the 3-second target, cost per request, retrieval hit rate and eval scores exported as metrics, alerted on SLA-agreed thresholds.
- 05 · DecideFortnightly sprint · monthly reportTuning runs against the edge cases each two-week cycle produced; the monthly report on quality, spend and edge cases goes to the business.
Blocked releases, grounded checks & drift on the inputs
A regression can't ship quietly
Every prompt or model change replays the frozen evaluation set first, and a regression blocks the release instead of opening a ticket someone might read later.
Hallucination checks are grounded
Where possible an assertion is verified against the source the system retrieved, so the detector isn't another model's opinion of the answer.
Drift shows before answers degrade
The input distribution is watched alongside output quality. The incident that started this went unnoticed for 11 days because only uptime was monitored.
Would you know if your AI’s answers got worse last week? Scope your build in 3 minutes.
Scope your buildNearby engagements
Data & AnalyticsA checkout that stopped losing sales to a 6-second load
Profiling found the real bottleneck behind a slow checkout (an N+1 query and an oversized bundle) and tuned both, with before-and-after metrics locked in as a baseline against future regressions.
E-commerce & Retail · 5 weeks
Web PlatformsA reader people finish, and a library that remembers where they stopped
A reading platform for a comics catalog: a browse surface people can actually navigate, a reader that gets out of the way, and a history that puts everyone back on the page they left.
E-commerce & Retail · 18 weeks
Web PlatformsOne number, a whole handset, and a database that keeps up with 125 brands
A metered IMEI lookup service that turns fifteen digits into a device, its specifications and its status. It's sold three ways to three audiences and backed by a device database that maintains itself.
Consumer Electronics · 22 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
