The double-charge bug that only ever showed up in production
A rare double-charge complaint traced back to a race condition between two webhook deliveries: reproduced from production logs, fixed, and locked down with a regression test.
The shape of the work
Professional Services
3 weeks
Fixed price
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
What went wrong, and when
- 01
The bug never appeared in staging or on-demand testing, only for a small fraction of real customers under real traffic, which meant two prior attempts to fix it had shipped changes that didn't actually address the cause.
It couldn't be reproduced locally, in staging, or on demand, because the trigger was two webhook deliveries of the same event arriving on different instances within a few milliseconds of each other, a condition only real traffic produces. Both prior fixes had targeted plausible causes and neither had touched the actual one.
We reconstructed the failure from production webhook logs, found that two near-simultaneous webhook deliveries for the same event could both pass the idempotency check before either had written its result, fixed the race with a proper lock, and wrote a regression test that fires both webhooks concurrently to prove it can't happen again.
Phase by phase
Phase 1: Gather
Every reported case
Pulled production logs for every double-charge complaint and looked for the shared pattern.
- Incident log set
- Pattern analysis
Phase 2: Reconstruct
Timing, precisely
Reconstructed the timing of concurrent webhook deliveries against the idempotency check.
- Timing reconstruction
- Reproduction script
Phase 3: Isolate
The window between check and write
Isolated the race: both deliveries passed the check before either had written, because the check and the write weren't atomic.
- Root-cause writeup
Phase 4: Fix and prove
Lock, then test concurrently
Fixed the race with a proper lock and added a regression test firing concurrent webhooks.
- Lock implementation
- Concurrent regression test
The casebook: gather, reconstruct, isolate, fix and prove; eighteen months of reports and two undiagnosed fixes, the three-week engagement with the root cause on day four, and zero reports since.
The numbers, before and after
Recurring → zero since fix
Double-charge reports
18 months of intermittent reports → 4 days to isolate
Root cause found
Concurrent webhook race now tested
Regression coverage
The double-charge figure is a count of customer reports, recurring before the fix and zero in the two quarters since. Time to root cause is measured from the engagement starting, against eighteen months of intermittent reports before it. Regression coverage is a statement about the test, which fires both deliveries concurrently against a real database.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
The engagement
There had been a handful of customer complaints about being charged twice for the same invoice. Rare, unreproducible locally, and dismissed twice before as a one-off billing-provider glitch.
A billing integration that had produced a handful of double-charge complaints over eighteen months: rare enough to be dismissed as a provider glitch, and dismissed twice on exactly that basis. The engagement was three weeks and started from the position that two previous fixes had shipped without a diagnosis, so the first job was to stop guessing.
Production Diagnosis & Bug Fixing
How it was handled
- 01
Pulled production logs for every reported double-charge to find the common pattern
Nothing was changed until the failure had been reconstructed from evidence, because two previous fixes had already been spent on plausible causes.
- 02
Reconstructed the timing of concurrent webhook deliveries against the idempotency check
Read as a timeline, the records showed two deliveries of one event eleven milliseconds apart on different instances, invisible in either request alone.
- 03
Isolated the race condition: two webhooks both passing the check before either wrote its result
Isolating it took four days of the three weeks, and confirmed that both previous fixes had targeted plausible causes and missed the real one.
- 04
Fixed the race with a proper lock and added a regression test firing concurrent webhooks
An advisory lock on the event ID now spans the whole handler, and the regression test fires both deliveries concurrently against a real database.
Reproduced from logs
The failure reconstructed from production webhook timing, with nothing left to guesswork.
The bug had no reproduction, so the logs were the evidence. Structured webhook records were aggregated and sorted by provider delivery ID, which showed two deliveries of the same event arriving eleven milliseconds apart on different instances. That interleaving was the whole failure, and it wasn't visible in any single request's log. Only in the two side by side.
- Reconstructed from aggregated production logs, not a repro
- Two deliveries of one event, eleven milliseconds apart
- Only visible with both requests correlated side by side
The two deliveries laid over one another from aggregated production logs: the same event on two instances, eleven milliseconds apart, both holding a “not seen” answer while each logged a clean success.
The real race
How each event is guarded and what is locked right now: an advisory lock on the event ID held across the whole handler, a second delivery waiting its turn, and the handler diff that put check and write in one critical section.
Two concurrent deliveries both clearing the idempotency check before either wrote its result.
The idempotency check read the processed table and wrote to it after the work completed, leaving a window in which both deliveries read empty and both proceeded. It was replaced with a Postgres advisory lock taken on the event ID for the duration of the handler, so the second delivery blocks, then finds the record written and exits. The check and the write now sit inside one critical section.
- Read-then-write window was the actual defect
- Advisory lock on the event ID held across the whole handler
- Check and write moved inside one critical section
A concurrent regression test
A test that fires both webhooks simultaneously, so the bug can't return quietly.
The regression test fires both deliveries concurrently against a real database instead of asserting the lock is called, because a mock proves the code was written, not that the race is closed. It runs a hundred iterations in CI and asserts exactly one side effect each time. It's the kind of test that fails loudly if someone later moves the lock for a good-sounding reason.
- Both webhooks fired concurrently against a real database
- Asserts exactly one side effect across a hundred iterations
- Would fail if the lock is later moved or narrowed
The regression test that fires both deliveries at once against a real database, a hundred iterations per CI run with exactly one side effect each, and the manual runs against the old handler that fail.
Working inside their operation
- 01
A cross-functional team of 3 worked on a fixed price basis over 3 weeks, covering Production diagnosis, Root-cause fix, Regression testing. We ran daily standups with their own lead in the room, and a demo at the end of every sprint. Scope changed twice during the engagement, and both times the change was priced and agreed before work started.
Three weeks, fixed price, and structured as a diagnosis before any fix: nothing was changed until the failure had been reconstructed from evidence. Production logs were aggregated and read as a timeline instead of a list of errors, and that's the whole difference. The fault is invisible in any single request and obvious with two side by side.
What changed in the runbook
- 01
An idempotency check that isn't atomic with its write isn't an idempotency check.
The check and the write have to be one critical section. Separated by any amount of work, they describe an intention and enforce nothing.
- 02
The bug was unreproducible until the logs were read as a timeline instead of as errors.
Errors were the wrong lens: each request logged a clean success, and the fault only exists in the relationship between two of them.
- 03
A regression test that doesn't run the operations concurrently would have passed against the broken code.
A test asserting the lock is called would have passed against the broken code. Only running the operations concurrently tells the two apart.
11 milliseconds was enough to charge a customer twice
The same pair of deliveries through the old handler and the fixed one. Step through the interleaving and watch the lock, the processed table and the charge count. Switch tabs with the arrow keys once one is focused.
The v1 handler: check the processed table, charge, then write the row once the work is done.
- In flight…
- In flight…
The billing provider delivers ev_7QK2M9TD4X twice. Both deliveries land on different instances.
From a duplicate delivery to exactly one charge
The provider can still deliver an event twice. What changed is everything after the delivery lands: the evidence that found the race, the lock that closes it, and the test that keeps it closed.
- 01 · TriggerProvider webhook deliveriesTwo deliveries of one event can arrive 11 ms apart on different instances, a condition only real traffic produces.
- 02 · IngestStructured logs, aggregatedWebhook records aggregated across instances and sorted by provider delivery ID, so two requests read as one timeline.
- 03 · EngineAdvisory lock on event IDA Postgres advisory lock held for the whole handler; the check and the write sit inside one critical section.
- 04 · StateProcessed recordWritten inside the lock, so a second delivery blocks, then finds the record and exits without a side effect.
- 05 · GuardConcurrent regression testBoth deliveries fired at once against a real database, 100 iterations in CI, exactly one side effect each time.
Evidence, a real lock & a concurrent guard
Diagnosed before anything changed
Nothing was touched until the failure had been reconstructed from production logs. Two earlier fixes had shipped against plausible causes; this one shipped against the evidence.
One charge per event, enforced
An advisory lock on the event ID spans the whole handler, with the check and the write in one critical section. A second delivery waits, finds the record, and exits.
The race can't return quietly
A test fires both deliveries concurrently against a real database, 100 iterations in CI, asserting exactly one side effect. Moving or narrowing the lock fails it.
Is a bug nobody can reproduce still reaching your customers? Scope your build in 3 minutes.
Scope your buildNearby engagements
AI & AutomationA private legal assistant grounded in verified precedents
A private knowledge assistant that searches internal case files and precedents, providing cited answers legal teams can verify in seconds.
Legal & Law Firms · 14 weeks
Product DesignAn onboarding flow that guides trial users to value
A redesigned SaaS trial onboarding experience with progressive checklists, sample data, and inline guidance that turns signups into active subscribers.
Professional Services · 10 weeks
Product DesignA design system that brought speed and consistency to 4 product teams
A token-based design system in Figma and React that eliminated component duplication across 4 product squads and cut the time from design handoff to merged frontend.
Professional Services · 14 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
