From a Thursday-evening deploy ritual to shipping 40+ times a week
A containerized build and a gated CI/CD pipeline replaced a manual, once-a-week deploy ritual, with a one-click rollback that turns a bad release into a non-event.
Who, what, and how long
- Industry
- Professional Services
- Duration
- 7 weeks
- Cooperation model
- Fixed price, phased
- Services
- ContainerizationCI/CD pipelineRollback strategy
- Integrations
- HubSpotDocuSignXeroGoogle Workspace
- Technologies
- DockerGitHub ActionsTerraformContainer registryBlue-green deployment
- Team
- 1 Project lead2 Frontend engineers1 Backend engineer
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
What went wrong, and when
Deploys ran on undocumented tribal knowledge, environments drifted from each other, and the only rollback plan was redeploying an old build by hand and hoping it still worked.
Environments were built independently and had drifted, so staging didn't predict production: a class of failure the team had learned to expect and couldn't diagnose. Deploy steps lived in one engineer's head. And the rollback path had never been exercised, so nobody knew whether the old build would still run against the current database.
We containerized the app into a reproducible image, built a CI pipeline that tests and gates every commit, and wired automated delivery that promotes one artifact through environments with a one-click rollback if anything goes wrong.
Phase by phase
Phase 1: Containerise
Reproducible builds
Packaged the application into a reproducible image so a build stopped depending on the machine it ran on.
- Dockerfile
- Build pipeline
Phase 2: Gate
Nothing merges unchecked
Built automated build, test, and lint stages gating every merge.
- CI stages
- Branch protection
Phase 3: Promote
One artifact, three environments
Wired artifact promotion from staging to production so the tested thing is the shipped thing.
- Promotion pipeline
- Environment config
Phase 4: Prove rollback
Under a real failure
Set up a one-click rollback and verified it by deliberately shipping a broken build.
- Rollback path
- Verification report
- Release runbook
The tail of the week's releases, runs 1,476 to 1,482, with the rolled-back renderer bump among six clean ones, beside the weekly digest: 41 deploys against one a week before, nine minutes from commit to production, and integration tests as the 40% that goes next.
The numbers, before and after
1 → 40+
Deploys per week
Hours → under a minute
Rollback time
0
Weekend on-call deploys
Deploys per week is the pipeline's own count in the quarter after cutover. Rollback time is measured from the decision to roll back to the previous version serving traffic, taken from the deliberate failure and from two real incidents since. Weekend on-call deploys is a count, and it's zero because the window no longer exists.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
The engagement
Deploys happened manually once a week, after hours, with an engineer on standby in case something broke. The ritual made shipping feel risky and held every feature to the weekly window.
A team shipping once a week, on Thursday evening, with an engineer on standby. The ritual was rational: the only rollback was redeploying an old build by hand, so every deploy was a one-way door and the only defense was doing it rarely with someone watching. The engagement was commissioned to remove the door, not to deploy more often.
DevOps & Release Engineering
How it was handled
Packaged the app into a reproducible Docker image
One image is built per commit and promoted through environments unchanged, with configuration injected at run time instead of baked in per environment.
Built automated build, test, and lint stages gating every merge
Build, test and lint run as required checks that block the merge, and Terraform plans post the infrastructure diff into the same review.
Wired artifact promotion through staging to production
Promotion went in before automated deployment, so the team had a working manual path through the pipeline before anything ran without them.
Set up a one-click rollback path and verified it under a real failure
The rollback was verified by deliberately shipping a broken build to production in a quiet window, because an unexercised path is a hypothesis.
One reproducible artifact
A container image built once and promoted through environments unchanged.
One container image is built per commit, tagged with that commit, and promoted through staging and production unchanged. Nothing is rebuilt per environment, because a rebuild is a new artifact however identical the Dockerfile claims it is. Configuration arrives as environment variables at run time, so the thing tested is byte-for-byte the thing that ships.
- Built once per commit, promoted unchanged
- No per-environment rebuilds. A rebuild is a new artifact
- Configuration injected at run time, never baked in
The nightly drift compare: one image digest across development, staging and production, the same run-time configuration keys and no baked-in config, with the only difference three feature flags meant to be ahead during a build.
A change blocked before it reaches main: build, unit specs and lint green, a failing integration test stopping the merge, and the Terraform plan posted into the same review with a reviewer's question on it.
Gated merges
Build, test and lint stages block a merge up front, before anything reaches main.
Build, unit tests, integration tests and lint run as required status checks on the pull request, so a failure blocks the merge instead of arriving as a notification after main is already broken. Terraform plans run on the same trigger and post the infrastructure diff into the review, which stopped configuration changes landing without anyone reading what they'd actually do.
- Required checks block the merge; nothing reports after the fact
- Terraform plan diff posted into the review
- Infrastructure changes reviewed as part of the code change
Rollback as routine
The bad release end to end: a 5xx spike on statement PDFs at 9:57 AM, one click by the engineer on call, and production back on the previous run 34 seconds later, served from the slot that was kept warm.
A one-click rollback path, proven under a real failure.
Blue-green means the previous version is still running and still healthy when the new one takes traffic, so rollback is a routing change: seconds, where a rebuild would cost ten minutes. It was tested by deliberately shipping a broken build to production during a quiet window, because a rollback path that's never been exercised is only a hypothesis.
- Blue-green: rollback is a routing change, not a rebuild
- Previous version kept warm and healthy
- Verified by shipping a deliberately broken build in a quiet window
Working inside their operation
A cross-functional team of 3 worked on a fixed price, phased basis over 7 weeks, covering Containerization, CI/CD pipeline, Rollback strategy. We held a standing mid-week checkpoint and wrote decisions down in place of status meetings. Nothing shipped that they hadn't seen working first.
Seven weeks, phased so the gate landed before the automation: nothing deployed automatically until build, test and lint were blocking merges, which meant the team spent two weeks with a pipeline that only refused things. That order was deliberate. A team that trusts the checks will accept automated deployment, and one that doesn't will route around it.
What changed in the runbook
The weekly deploy ritual was a symptom of having no way back. Deploying too often was never the problem.
Deploy frequency was a consequence, never the goal: the team shipped weekly because that was the safe rate for a one-way door, and the rate rose on its own once there was a way back.
Promoting one artifact removed a whole class of works-in-staging failures.
The works-in-staging failures were environment drift, and promoting one artifact removes drift by construction. Nobody has to keep two builds in agreement.
A rollback nobody has exercised isn't a rollback. The deliberate failure was the point.
The deliberate failure is what turned the rollback from a documented procedure into a rehearsed one, and that's the difference at two in the morning.
One bad release, replayed
Why the way back takes seconds and never a rebuild
The renderer bump that broke statement PDFs, step by step: where each run was, which slot held the traffic, and what one click actually changed. Switch tabs, or use the arrow keys once one is focused.
Run 1,479 · Thu 09/17/2026 · blue-green slots A and B
9:58:12 AMOne click points the router back at slot A. Nothing is built, pulled or started, because run 1,478 never stopped. It is serving all traffic at 9:58:46 AM, 34 seconds after the decision; rebuilding it would have cost about 10 minutes.
- Click confirmed, decision logged~3 s
- Router pointed at slot A~6 s
- In-flight requests on slot B drain~21 s
- Slot A verified serving~4 s
From a commit to production traffic, with the way back kept warm
The gate went in before the automation: for two weeks the pipeline only refused things, and nothing deployed on its own until build, test and lint were blocking merges.
- 01 · TriggerPull request / mergeEvery commit runs the pipeline; Terraform plans run on the same trigger and post the infrastructure diff into the review.
- 02 · GateBuild, test, lintRequired status checks block the merge, before main can break.
- 03 · ArtifactOne image per commitTagged with its commit and never rebuilt per environment, because a rebuild is a new artifact.
- 04 · PromoteStaging → productionThe same image moves through environments unchanged; configuration arrives as environment variables at run time.
- 05 · ServeBlue-green routingThe previous version stays running and healthy, so rollback is a routing change: seconds, where a rebuild takes ten minutes.
So a bad release stays a non-event
Blocked early, identical everywhere, reversible in seconds
Broken code stops at the merge
Build, unit tests, integration tests and lint are required checks on the pull request, so a failure blocks the merge instead of arriving as a notification after main is broken. Terraform plans post their diff into the same review.
Staging predicts production
One image is built per commit and promoted unchanged, with configuration injected at run time. The thing tested is byte-for-byte the thing that ships, which removes environment drift by construction.
A way back that has been used
The previous version stays running and healthy, so rollback is a routing change. It was proven by shipping a deliberately broken build in a quiet window, and has since been timed in two real incidents.
Is every deploy still a one-way door? Scope your build in three minutes.
Scope your buildNearby engagements
AI & AutomationA private legal assistant grounded in verified precedents
A private knowledge assistant that searches internal case files and precedents, providing cited answers legal teams can verify in seconds.
Legal & Law Firms · 14 weeks
Product DesignAn onboarding flow that guides trial users to value
A redesigned SaaS trial onboarding experience with progressive checklists, sample data, and inline guidance that turns signups into active subscribers.
Professional Services · 10 weeks
Product DesignA design system that brought speed and consistency to 4 product teams
A token-based design system in Figma and React that eliminated component duplication across 4 product squads and cut the time from design handoff to merged frontend.
Professional Services · 14 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
