The release gate that turned a green pipeline into a real promise
A layered automated test suite and a CI release gate replaced a shipping process built on crossed fingers, and a flake-cleanup pass made a red result mean something again.
What the engagement involved
- Industry
- Professional Services
- Duration
- 6 weeks
- Cooperation model
- Fixed price
- Services
- Test suite designRelease gatesFlake management
- Integrations
- HubSpotDocuSignXeroGoogle Workspace
- Technologies
- PlaywrightVitestTypeScriptGitHub ActionsTest containersCoverage reporting
- Team
- 1 Project lead2 Frontend engineers1 Backend engineer
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
What went wrong, and when
The team had tried automated tests twice before, and both suites had rotted into a wall of flaky failures everyone learned to ignore. By the time we arrived, "run the tests" had stopped meaning anything.
Both previous suites had chased a coverage percentage, which spreads effort evenly across code whose failure costs are anything but even. They were also slow and intermittent, so a red build got re-run instead of investigated. By the time we arrived, "run the tests" had stopped carrying information, and the release decision had quietly reverted to whoever was on call.
We mapped the quoting and billing paths that would hurt most if they broke, built a layered suite of unit, integration, and full-flow tests around them, wired a CI release gate that blocks a red build from shipping, and quarantined or fixed every flaky test until green reliably meant safe.
Phase by phase
Phase 1: Map risk
What hurts if it breaks
Mapped the quoting and billing flows carrying the most regression risk, and deliberately left coverage elsewhere thin.
- Risk map
- Coverage targets
Phase 2: Build layers
Unit to full flow
Built layered unit, integration, and end-to-end tests aimed at those specific flows.
- Test suite
- Fixtures & factories
Phase 3: Gate
Blocking, not advisory
Wired a CI release gate that blocks a failing build from deploying at all.
- Release gate
- CI configuration
Phase 4: De-flake
Make red mean red
Audited the existing suite, quarantined flaky tests, and fixed their root causes instead of retrying them.
- Flake audit
- Quarantine list
- Root-cause fixes
Every suite, its layer and what it costs to run: Vitest units, integration against Postgres in test containers, and Playwright flows through quote, bind and invoice.
The numbers, before and after
0 → 14 in first month
Regressions caught pre-release
31% → under 2%
Flaky test rate
Retired
Manual pre-release click-through
Regressions caught pre-release is the count in the first month after the gate went live, against zero before, because there was no mechanism to catch them. Flaky rate is the share of runs failing without a code change, measured over rolling two-week windows. The manual click-through is retired, which is a state, not a measurement.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
The engagement
Quote and billing changes shipped with no automated coverage at all. Every release rode on a manual click-through and whichever bugs the on-call engineer happened to notice first.
An insurance platform where quoting and billing changes shipped on a manual click-through, and where two previous automated suites had already rotted into walls of flaky failures the team had learned to ignore. So the engagement had two jobs, and the second (making green mean something again) was the harder and the more important.
Quality Engineering
How it was handled
- 01
Mapped the quoting and billing flows carrying the most regression risk
The business signed off on the risk map, since what a failure costs is a commercial call first and a technical one second.
- 02
Built layered unit, integration, and end-to-end tests targeted at those flows
Three layers, each doing what it's good at: units for the premium edge cases, integration against a real database, and full flows through quote, bind and invoice.
- 03
Wired a CI release gate that blocks a failing build from deploying
The gate went live only after the flaky rate was under control, because a blocking check on an unreliable suite would have been overridden within a week.
- 04
Audited the existing suite, quarantined flaky tests, and fixed their root causes
Every intermittent test was quarantined out of the gate on first sighting and root-caused, which is what made the pass informative again.
Risk-targeted coverage
Tests concentrated on the quoting and billing paths that would hurt most if they broke.
Coverage was aimed, not maximized. The quoting and billing paths were mapped for what a failure would actually cost (a wrong premium, a double charge, a policy issued without payment), and tests were concentrated there. Peripheral code has thin coverage on purpose: a suite that treats every line as equally important takes hours to run and tells you nothing about risk.
- Failure cost mapped before a single test was written
- Depth concentrated on quoting and billing
- Peripheral code deliberately thin, so the suite stays fast
Coverage by module against a risk map agreed with the business: modules that move money held to the deepest target, peripheral code deliberately thin, and the three modules sitting under their threshold.
A red build: an integration test against real Postgres finds twelve installments that no longer sum to the premium, the change that caused it, and a deploy held by a required check with no override.
A gate that blocks
A red build can't deploy. No noting it and overriding it.
Three layers, each doing what it's good at: Vitest units for the premium calculation's edge cases, integration tests against a real Postgres in a test container (no mock that agrees with itself), and Playwright flows through quote, bind and invoice. The gate is a required check, so a red build can't deploy. Before, it was a status somebody could override, and did.
- Units, container-backed integration, and full Playwright flows
- Integration tested against real Postgres
- Required check: a red build can't be overridden
Green that means something
The flake register: each intermittent test quarantined out of the gate on first sighting, ticketed, root-caused (mostly unawaited async and shared fixtures), then fixed or deleted.
Flaky tests quarantined and root-caused, so a pass is informative again.
Flakiness was treated as a defect, never as weather. Every intermittent test was quarantined out of the gate immediately, root-caused, and either fixed or deleted; most were racing on unawaited async work or on shared fixture state. A suite that fails one time in ten teaches a team to re-run it, and a gate nobody believes stops being a gate.
- Flaky tests quarantined out of the gate on first sighting
- Root-caused to unawaited async and shared fixtures
- Fixed or deleted, never left rerunning until green
Working inside their operation
- 01
A cross-functional team of 3 worked on a fixed price basis over 6 weeks, covering Test suite design, Release gates, Flake management. We held a standing mid-week checkpoint and wrote decisions down in place of status meetings. Nothing shipped until they'd seen it working.
Six weeks, fixed price, with the risk map produced first and agreed with the business, because what a failure costs is a commercial question. Flakiness was treated as a defect from the first sighting: quarantined out of the gate immediately, root-caused, then fixed or deleted, never left re-running until green.
What changed in the runbook
Coverage percentage was the wrong target. The risk map was the right one.
Coverage percentages treat every line as equally important, which is false and expensive. The risk map put depth where a failure costs money and left the rest deliberately thin.
A flaky suite is worse than no suite: it teaches the team to ignore a red build.
A suite that fails one time in ten teaches a team to re-run instead of investigate, and after that, only removing the flakiness can repair it. More tests won't.
The gate only had authority because the tests had stopped lying.
The gate is a social object as much as a technical one. It only holds because the team believes a red build means something, and that belief is what the flake work bought.
Three builds, one gate
Why one build ships, one is held, and one flaky failure doesn’t count
Pick a build and the gate checks its suites, the flake quarantine and risk-tier coverage, then open any criterion to see why it passed or held. Switch tabs, or use the arrow keys once one is focused.
A build at the gate
Against a real Postgres, twelve instalments of $106.99 sum to $1,283.88, not the $1,283.99 premium: the PR dropped the remainder. A mock that agreed with itself would have passed. The check is required, so there's no override. Fix and push.
From a pull request to a deploy the gate has allowed
The quarantine sits in front of the gate on purpose: noise is taken out before the result is read, so what stays red is a real regression and what stays green is safe to ship.
- 01 · TriggerPull request on GitHub ActionsEvery quoting and billing change runs the suite; nothing ships on a manual click-through anymore.
- 02 · SuitesVitest · Test containers · PlaywrightUnits for premium edge cases, integration against a real Postgres, flows through quote, bind and invoice.
- 03 · QuarantineFlake registerA test failing without a code change leaves the gate on first sighting, is root-caused, then fixed or deleted. It's never re-run until green.
- 04 · GateRequired release checkA red build can't deploy. The check is required, so nobody can note it and override it.
- 05 · DeliverDeploy on green onlyThe deploy runs only once the gate passes, and the gate went live only after the flaky rate was under control.
So a billing mistake stays in CI
A gate the team believes
A red build cannot deploy
The gate is a required check, so a failing build is held. The old status let people note a failure and override it, and they did.
Tested against a real database
Integration tests run against a real Postgres in a test container, and Playwright flows go through quote, bind and invoice, so billing is never checked by a mock that agrees with itself.
Nobody learns to re-run
A flaky test is quarantined out of the gate on first sighting, root-caused, then fixed or deleted. With the flaky rate down from 31% to under 2%, a red result gets investigated instead of retried.
Does a green build actually mean your release is safe? Scope your build in 3 minutes.
Scope your buildNearby engagements
AI & AutomationA private legal assistant grounded in verified precedents
A private knowledge assistant that searches internal case files and precedents, providing cited answers legal teams can verify in seconds.
Legal & Law Firms · 14 weeks
Product DesignAn onboarding flow that guides trial users to value
A redesigned SaaS trial onboarding experience with progressive checklists, sample data, and inline guidance that turns signups into active subscribers.
Professional Services · 10 weeks
Product DesignA design system that brought speed and consistency to 4 product teams
A token-based design system in Figma and React that eliminated component duplication across 4 product squads and cut the time from design handoff to merged frontend.
Professional Services · 14 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
