An order pipeline that scales for month-end and costs nothing at 2am
An event-driven serverless rebuild of a B2B order-processing pipeline: triggered per purchase order, scaling for month-end volume, and costing next to nothing overnight.
The shape of the work
- Industry
- Logistics & Distribution
- Duration
- 10 weeks
- Cooperation model
- Fixed price
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
The hard problem
A provisioned server cluster processed incoming EDI and API purchase orders continuously, sized for the handful of month-end days when volume actually peaked. Idle capacity was paid for most nights, and a promotion that pushed volume past what the cluster could absorb still risked a backed-up queue.
Validation, pricing and fulfillment routing ran in one process, so they scaled as one, and pricing called a third-party service that was slower than the other two combined. Under load the whole pipeline ran at pricing's pace, which meant adding capacity mostly bought more concurrent calls to a vendor that then rate-limited them.
We decomposed the order pipeline into functions triggered directly off each incoming order, fanned out through a managed queue to validation, pricing, and fulfillment-routing steps, and moved order state into managed storage so every function stayed stateless and could scale independently.
How the pieces fit
- 01
Mapped the pipeline into discrete functions: intake, validation, pricing, fulfillment routing
The pipeline was mapped into stages by what each actually waits on, which is how pricing's third-party call turned out to be setting the pace for everything.
- 02
Replaced the provisioned server cluster with event-triggered functions and a managed queue
An incoming order raises an event and starts the pipeline on that order alone, fanned out through per-stage queues each with its own failure domain.
- 03
Tuned concurrency and cold-start handling for the month-end volume spike
Tuning happened against a replay of a real month-end instead of synthetic load, because the shape of the spike is what breaks things, not its size.
- 04
Moved order state into managed storage so functions stayed stateless
Order state moved to managed storage with conditional writes, so a redelivered message re-reads, sees the work done and exits instead of pricing twice.
The system we were asked to build
A fixed cluster of servers ran around the clock to process incoming purchase orders, sized for the volume spike at month-end close. We rebuilt the pipeline as event-triggered functions backed by a managed queue and managed storage, so it scales with real order volume instead of a server count fixed for the worst week of the month.
A distributor taking purchase orders over EDI and an API, processed by a server cluster sized for the three days at month end when volume peaks. That cluster ran every night at a few percent utilization. The engagement was commissioned after a promotion pushed volume past the cluster's ceiling on an ordinary Tuesday and orders queued for six hours before anyone noticed.
Cloud-Native Architecture
Triggered per order
Functions fire on each incoming purchase order instead of a cluster polling for work.
The old cluster polled for work and sat idle between peaks. Now an incoming purchase order raises an EventBridge event and the pipeline starts on that order alone, so latency doesn't depend on where in the polling cycle it arrived. Fan-out through SQS gives each stage its own queue and its own failure domain, and a poison message parks in a dead-letter queue instead of stalling the line.
- EventBridge trigger per order: no polling cycle to wait on
- Per-stage SQS queues, each its own failure domain
- Poison messages park in a DLQ instead of stalling the line
One purchase order traced through every hop on one clock: started by its own event, rejected on validation, retried after backoff, and acknowledged by the ERP 38.8 seconds after it arrived.
Month-end capacity: 3,110 orders at the peak hour, above the old cluster's ceiling, and concurrency set per function, with pricing held at its reserved limit so the spike queues at that step alone.
Independent scaling
Validation, pricing, and routing scale separately, so the slowest step stops throttling the rest.
Validation is cheap and bursty, pricing calls a slow third party, and fulfillment routing is heavy but rare. As one process they ran at the speed of the slowest; as separate functions with their own concurrency limits they no longer contend. Pricing gets a reserved concurrency that respects the vendor's rate limit, so a spike queues at that step instead of triggering vendor-side throttling for everyone.
- Per-function concurrency, so the slowest step stops throttling the rest
- Reserved concurrency on pricing to respect a vendor rate limit
- Spikes queue at one step while the rest of the line keeps moving
Stateless by design
The traced order's state in DynamoDB and every write against it. Each is conditional on the state the function expected, so the redelivered pricing message re-read, found the order priced and exited.
Order state lives in managed storage, so any function can be retried or scaled without coordination.
Order state lives in DynamoDB keyed by order, so no function holds anything between invocations and any of them can be retried, replaced or scaled without coordination. Writes are conditional on the state the function expected to find, which makes retries safe: a duplicate delivery from the queue re-reads, sees the work already done and exits instead of pricing the order twice.
- State in DynamoDB; functions hold nothing between invocations
- Conditional writes make queue redeliveries safe
- Any function retried, replaced or scaled without coordination
Phase by phase
Phase 1: Decompose
Four steps, not one process
Mapped the monolithic pipeline into discrete stages (intake, validation, pricing, fulfillment routing) with the contract between each written down.
- Pipeline map
- Stage contracts
Phase 2: Rebuild
Events instead of a cluster
Replaced the provisioned cluster with event-triggered functions fanned out through a managed queue.
- Function set
- Queue topology
- IaC definitions
Phase 3: Tune
Month-end without a scramble
Tuned concurrency limits and cold-start behavior against the real month-end volume curve, not an average.
- Load test results
- Concurrency configuration
Phase 4: Externalise state
Nothing held in memory
Moved order state into managed storage so every function stayed stateless and independently retryable.
- State schema
- Retry & idempotency design
The retry policy of five attempts with backoff, and the three orders held for replay out of 41,184, each with its original payload kept and its error stated in plain words.
What it carries now
−92%
Overnight compute cost
0
Month-end order backlog
0
Servers to patch
Overnight compute cost compares the month after cutover with the month before, on the same order volume. Month-end backlog is a count of orders unprocessed at the close of the peak window. Servers to patch is literal: there are none, a secondary goal that turned out to matter more than expected to the operations team.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
What the architecture settled
- 01
Month-end was never a capacity problem. It was a coupling problem between stages.
The cluster had been sized for month end because the stages couldn't scale apart. Once they could, the peak was three independent problems, and none of them was hard.
- 02
Statelessness is what made the retries safe; without it, scaling just multiplied the failure modes.
Retries are only safe when the function holds nothing between invocations. A stateful retry at scale multiplies the failure modes instead of absorbing them.
- 03
The overnight bill going to almost nothing was a side effect of the right decomposition, not the goal.
Paying almost nothing overnight followed from stages that scale to zero independently: a consequence of the decomposition more than its objective.
How the work was run
A cross-functional team of 4 worked on a fixed price basis over 10 weeks, covering Serverless architecture, Event-driven pipeline, Managed data & queues. We ran two-week increments, each one shippable, reviewed with them before it merged. Decisions were recorded as they were made, so the reasoning survived the people who made it.
Ten weeks, fixed price, with the pipeline rebuilt beside the cluster and orders mirrored through both until the outputs matched for a full month-end. Infrastructure was defined as code from the first commit, which is what made running two pipelines in parallel a configuration change instead of a project.
One purchase order, hop by hop
Rejected once, retried, priced once, and in the ERP 38.8 seconds after it arrived.
Follow a single order through intake, its queue, validation, pricing, routing and dispatch. Each hop shows what it waited on, what happened and what it wrote to the order’s state. Switch hops, or use the arrow keys once a tab is focused.
- 06:22.5Attempt 1 of 5 startsRunning
- 06:23.1Rejected: delivery point 5060412380017 not in the partner directory yetRejected
- 06:23.1Message back on validate-q, visible again in 8 sWaiting
- 06:31.1Attempt 2 of 5 startsRunning
- 06:31.6Valid: delivery point found, 18 of 18 lines passDone
Why a rejection didn't hold anything up: the failed message waited on its own stage’s queue while other orders kept moving, and because the order’s state lives in storage rather than in the function, the retry simply started again from what was written. Timings per hop are illustrative; the 38.8 s total, the rejection and the retry are the traced order’s.
From one purchase order to the ERP, with nothing kept warm
The provisioned cluster became event-triggered functions behind a managed queue and managed storage, defined as code from the first commit and run beside the old cluster until the outputs matched for a full month-end.
- 01 · TriggerEventBridge event per orderAn EDI or API purchase order raises its own event, so the pipeline starts on that order at once instead of waiting on a polling cycle.
- 02 · QueuePer-stage SQS queuesEach stage has its own queue and failure domain. A poison message parks in a dead-letter queue instead of stalling the line.
- 03 · EngineValidate, price, route in LambdaSeparate functions with their own concurrency limits. Pricing's reserved concurrency respects the vendor's rate limit, so a spike queues there.
- 04 · StateOrder state in DynamoDBKeyed by order and written conditionally. Functions hold nothing between invocations, so any of them can be retried or scaled without coordination.
- 05 · DeliveryDispatch to the ERPEvery stage scales to zero on its own between orders, which is why the overnight compute bill all but disappeared.
No lost orders, no double processing, no month-end backlog
A bad order parks; it doesn't stall the line
Each stage has its own queue and its own failure domain. A poison message moves to a dead-letter queue after five attempts with backoff, with its original payload kept for replay, while every other order keeps moving.
A redelivery can't price an order twice
Order state lives in DynamoDB and every write is conditional on the state the function expected. A duplicate delivery re-reads, sees the work already done and exits.
A spike queues at one step, not across the pipeline
Pricing runs under a reserved concurrency that respects the vendor's rate limit, so month-end volume waits at that step instead of triggering vendor-side throttling for everyone.
Paying all month for servers sized for three days of orders? Scope your build in 3 minutes.
Scope your buildNearby engagements
Data & AnalyticsWarehouse inventory that stays accurate on its own
An inventory automation layer that reconciles counts across warehouses, forecasts replenishment, and flags discrepancies before they become stockouts.
Warehousing & Fulfillment · 13 weeks
Web PlatformsOne long form is why the device records were empty, so the record became tabs
A small desktop tool for cataloging equipment, where a device record is split into tabs so a partial entry is a normal state, never an abandoned form.
Logistics & Distribution · 5 weeks
Web PlatformsA freight portal that brings visibility to every mile
A real-time shipment visibility platform connecting carriers, shippers, and dispatchers with live tracking, geofence alerts, and automated status updates.
Freight & Trucking · 18 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
