The outage where the homepage looked fine and checkout was dead
A checkout-specific health check caught a payment-provider integration failure within two minutes, while a shallow homepage ping would have shown everything as healthy for hours.
The shape of the work
E-commerce & Retail
Ongoing retainer
Ongoing retainer, phased
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
What went wrong, and when
- 01
A payment-provider integration failure took checkout down during a weekend sale while the homepage, product pages, and every shallow health signal stayed green, so nothing in place would have caught it.
The check measured the wrong thing entirely. A homepage ping proves a web server is answering, which is true in almost every commercially significant outage. Nothing exercised the payment path, so the class of failure that costs the most money was precisely the class the monitoring couldn't see. The outage was found by a customer email.
We defined the endpoints that actually represent a working store (homepage, product page, and a full checkout health check), wired alert thresholds tuned to fire on a real degradation, and set an escalation path so a fired alert reached the on-call engineer directly instead of sitting in an unread channel.
Phase by phase
Phase 1: Define healthy
What a working store means
Identified the endpoints that define a genuinely working store, well beyond a loading homepage.
- Endpoint inventory
- Health definition
Phase 2: Build the check
Exercise the real flow
Built a synthetic checkout health check that runs the actual payment path end to end.
- Synthetic check
- Test payment setup
Phase 3: Tune
Fire on degradation, not noise
Tuned thresholds so alerts fire on real degradation and stay quiet through normal traffic variance.
- Threshold configuration
- Noise analysis
Phase 4: Escalate
Straight to on-call
Wired an escalation path routing a fired alert directly to the on-call engineer.
- Escalation policy
- On-call rota
Incident 41, the weekend checkout outage: detected in under 2 minutes, on-call reached in under 5, about three hours of checkout downtime avoided (an estimate), with the first five minutes on a timeline and the minute-by-minute record.
The numbers, before and after
Under 2 minutes
Time to detect the outage
Under 5 minutes
Time to reach on-call
~3 hours of full checkout downtime avoided
Estimated sales saved during the sale
Time to detect and time to reach on-call are both from the first real incident under the new monitoring. The sales-saved figure is an estimate: the retailer's own average revenue per hour at that trading period applied to the downtime avoided, and it's presented as an estimate, not a measurement.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
The engagement
The only uptime check before we started was a basic homepage ping, so a broken checkout could sit undetected for hours as long as the homepage kept loading.
An online retailer whose entire uptime monitoring was a homepage ping. The engagement was commissioned after a weekend sale during which a payment-provider integration failed and checkout was down for a stretch while every signal in place stayed green, because the homepage, which doesn't touch payments, kept loading perfectly throughout.
Uptime & Reliability Monitoring
How it was handled
- 01
Identified the endpoints that define a genuinely working store, well beyond a loading homepage
The checks were rewritten around what a working store means (homepage renders, product page returns real stock, checkout completes) as three separate assertions.
- 02
Built a synthetic checkout health check that exercises the real payment flow
It runs against the payment provider in test mode on a dedicated SKU, so a real failure is caught without a real order ever being created.
- 03
Tuned alert thresholds to fire on a real degradation, not normal traffic noise
The thresholds were set from a month of existing traffic, because the previous alerts had fired often enough that everyone had learned to ignore them.
- 04
Wired an escalation path routing a fired alert straight to the on-call engineer
Alerts route to a named responder with an acknowledgment deadline and escalate to the next if it passes, instead of posting into a channel.
Checks that mean something
Endpoints chosen to represent a working store, beyond a homepage that happens to respond.
The old monitor pinged the homepage and reported green through an outage that stopped every sale. The checks were rewritten around what a working store actually means: the homepage renders, a product page returns real stock, and checkout completes. Three assertions instead of one, each attributed to a system, so an alert names the failing part instead of announcing that something is wrong.
- Homepage, product page and checkout checked separately
- Product check asserts real stock, not a 200 response
- Alerts name the failing system, not just the site
The health definition: homepage renders, product page returns real stock, checkout completes. Three assertions, each owned by a system and naming it in the alert, beside the retired homepage ping and the inventory of all nine checks.
One synthetic checkout run on the dedicated test SKU: product page, cart and address pass, the payment intent against the provider in test mode times out, confirmation is skipped and the test order is voided, while every page check at that moment stays green.
A synthetic checkout
A health check that exercises the real payment flow, end to end.
The checkout probe runs a real order end to end on a test SKU every few minutes (cart, address, payment intent, confirmation) against the live payment provider in test mode, then voids it. That's what catches the failures that matter: an expired API key or a declined provider doesn't affect any page a ping would ever request.
- A real order run end to end every few minutes
- Live payment provider in test mode, then voided
- Catches expired keys and provider failures a ping can't see
Escalation to a person
The checkout alert rule (two consecutive failures from two regions, thresholds from a month of traffic) with the six retired false-page rules, the four-step escalation chain with its acknowledgment deadline, the on-call schedule and the acknowledgment that closed the weekend page.
A fired alert routed to the on-call engineer, never left in an unread channel.
Alerts route to a person with an acknowledgment deadline and an escalation to the next responder if it passes, so a two-in-the-morning failure never sits in a channel until nine. Thresholds require two consecutive failures before paging, which removed the noise that had taught everyone to ignore the previous alerts in the first place.
- Paged to a named responder with an escalation deadline
- Two consecutive failures required before paging
- Noise removed, so the alerts are believed again
Working inside their operation
A cross-functional team of 5 worked on an ongoing retainer, covering Availability monitoring, Incident detection & alerting, Escalation & on-call. We ran a weekly demo and a shared board they could read at any time. Their team took over day-to-day operation of the checks while we stayed on the retainer for thresholds and escalation.
A phased retainer: checks first, then thresholds, then escalation. The escalation path was treated as part of the monitoring, never as an operational detail: the previous alerts had gone to a channel, and a channel at two in the morning is a place alerts go to wait. Thresholds require two consecutive failures before paging, which removed the noise that had trained everyone to ignore them.
What changed in the runbook
A homepage ping proves the web server is up and nothing else that matters.
Almost every outage that costs money leaves the web server answering, which is why the cheapest possible check is also the one that catches least.
The two-minute detection came from checking the flow that earns money, not the page that loads fastest.
Exercising the payment flow is what makes an expired key or a provider failure visible. Neither affects any page a ping would ever request.
An alert nobody receives is indistinguishable from no monitoring at all.
The escalation is the monitoring: detection at two minutes and delivery at nine in the morning is the same outcome as no detection at all.
Why the checkout failure was caught in under two minutes
Four moments from the weekend incident: a check fails, a second region confirms it, the alert rule fires, and a person is reached. The homepage stayed green through all of it. Switch tabs, or use the arrow keys once one is focused.
The page reaches a named person, who acknowledges before the deadline
- Paged
- T+1:33 · push + SMS
- Phone call
- T+3:33 · no ack yet
- Acknowledged
- T+4:12 · Under 5 min
The page went to the primary on-call engineer with a 5-minute acknowledgement deadline, not to a channel. Had it passed, the secondary would have been paged next; it didn't, so step 2 was never needed.
From a test order to the person who can fix it
The escalation is part of the monitoring, not an operational afterthought. A failure detected in two minutes and read at nine the next morning is the same outcome as no detection at all.
- 01 · DefineHealth definitionHomepage renders, product page returns real stock, checkout completes: three assertions, each attributed to the system that owns it.
- 02 · ProbeSynthetic checkoutA real order on a dedicated test SKU every few minutes, against the live payment provider in test mode, then voided.
- 03 · DecideAlert ruleTwo consecutive failures before paging, with thresholds set from a month of existing traffic, never guessed.
- 04 · RouteEscalation policyThe alert names the failing system and goes to a named responder, never into a channel where it waits until morning.
- 05 · ReachOn-call engineerAn acknowledgment deadline on every step; if it passes, the page escalates to the next responder in a four-step chain.
Real assertions, quiet thresholds, a person on the end
The payment path is exercised
The checkout probe runs a real order end to end against the live provider in test mode and voids it, so an expired key or a failing provider shows up, even though neither touches any page a ping would request.
Noise doesn't page anyone
Two consecutive failures are required before paging, and thresholds were set from a month of existing traffic. The false-page rules that had taught everyone to ignore alerts were retired.
A person is reached, or the next one is
Alerts go to a named on-call responder with an acknowledgment deadline and escalate to the next responder if it passes. A channel gets a copy, never the only copy.
Would your monitoring notice if checkout stopped taking payments? Scope your build in 3 minutes.
Scope your buildNearby engagements
Data & AnalyticsA checkout that stopped losing sales to a 6-second load
Profiling found the real bottleneck behind a slow checkout (an N+1 query and an oversized bundle) and tuned both, with before-and-after metrics locked in as a baseline against future regressions.
E-commerce & Retail · 5 weeks
Web PlatformsA reader people finish, and a library that remembers where they stopped
A reading platform for a comics catalog: a browse surface people can actually navigate, a reader that gets out of the way, and a history that puts everyone back on the page they left.
E-commerce & Retail · 18 weeks
Web PlatformsOne number, a whole handset, and a database that keeps up with 125 brands
A metered IMEI lookup service that turns fifteen digits into a device, its specifications and its status. It's sold three ways to three audiences and backed by a device database that maintains itself.
Consumer Electronics · 22 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
