Document AI that extracts the data, so nobody has to retype it
A document processing system that pulls tables, key terms and metadata out of scanned PDFs, and calls in a human reviewer only when a value is in doubt.
The shape of the work
- Industry
- Professional Services
- Duration
- 12 weeks
- Cooperation model
- Fixed price, phased
Client name withheld under NDA. Engagement details are shown to the extent our agreement permits.
What we were brought in to do
Staff spent hundreds of hours typing data from vendor invoices and contracts. We built a Document AI engine that extracts key data automatically.
A professional services firm processing around a hundred thousand vendor documents a month (invoices, contracts, statements) through a team whose entire job was reading them and typing the figures into a system. Roughly a third arrived as scans, some of them faxes. The engagement was commissioned when that team's headcount became the constraint on taking more clients.
AI & Automation
Where the old way broke
Scanned PDFs and inconsistent document formats forced staff to retype figures by hand, slowing operations and letting errors into the data.
The documents had no common format: two hundred vendors, each with their own layout, some changing it without notice. Plain OCR had been tried and failed on the scans, and the failure was silent. It returned confident text that was wrong, which is worse than returning nothing, because the errors reached the database and turned up in reconciliation weeks later.
We built a multi-stage Document AI pipeline combining OCR with LLM extraction, adding a visual exception review queue for low-confidence fields.
Layout-Aware OCR & Vision Pipeline
Extracts key-value fields and table data from scanned PDFs, images, and digital documents.
Pages go through Tesseract with layout analysis first, so tables come back as tables and a key-value pair keeps the geometric relationship that makes it one. The vision model reads the page image alongside the text, and that second read is what recovers fields from a bad fax that OCR alone loses. Every extracted value keeps the bounding box it came from.
- Layout-aware OCR: tables stay tables
- Vision model run alongside the text, as a second read
- Every value keeps the bounding box it was read from
A faxed invoice read twice: layout-aware OCR and a vision pass on the same page image, each value kept with its box, the line items returned as a table, and the one field where the passes disagree boxed in place.
The vendor invoice schema: twelve fields, each with a validator, a floor set by field type and a named fallback, bank details that never post themselves, and two dates scored from OCR confidence, agreement and their validator.
Confidence Scoring Engine
Assigns probability scores to extracted fields, auto-approving high-confidence data.
Confidence isn't the model's own token probability, which is a poor proxy. It combines OCR character confidence, agreement between the text and vision passes, and whether the value satisfies the field's own validator: a date that doesn't parse scores low however sure the model is. Fields above threshold sync automatically; the rest queue. The threshold is set per field type, never globally.
- Composite score: OCR, cross-pass agreement, field validation
- A value that fails its own validator scores low regardless
- Thresholds set per field type, never globally
Exception Review Workspace
A supplier statement in review: the page beside its extraction, column totals recomputed cleanly, and the one date a staple obscured highlighted where it sits, with the reviewer's keyboard shortcuts along the top.
Visual audit tool for human review of low-confidence entries before system sync.
The reviewer sees the page and the extraction side by side, with the disputed field highlighted in place on the original. The whole job is comparing two things, and a form that made you hunt for the source was why the previous queue was ignored. Corrections write back as training signal, and keyboard-only operation was a requirement for a queue worked all day, not a nicety.
- Page and extraction side by side, field highlighted in place
- Corrections captured as training signal
- Fully keyboard-operable, because it's worked all day
What we built together
- 01
Benchmarked extraction accuracy across historical document types
The benchmark set the target and, more usefully, identified which document types were hopeless and should route to a person by default.
- 02
Built a hybrid OCR and vision LLM extraction pipeline
Pages go through layout-aware OCR and a vision model on the same image, which is what recovers fields from a bad fax that text extraction alone loses.
- 03
Designed a reviewer UI for approving flagged exceptions
Confidence combines OCR character confidence, agreement between the two passes, and whether the value satisfies its own field validator. A date that won't parse scores low.
- 04
Connected structured output straight into internal databases
Structured output writes straight into the internal database with the source bounding box retained, so a disputed figure can be traced back to the pixel.
Operational results after launch
−85%
Data entry time cut
99.1%
Extraction accuracy
100k+
Monthly docs processed
Data entry time is the firm's own labor figure for the processing team, comparing the quarter after launch with the quarter before. Extraction accuracy is field-level, measured against a hand-labeled sample drawn after launch (not the benchmark set), and counts auto-approved fields only. The review queue is excluded because a human checked those.
Client name withheld under NDA. Figures are approximate, drawn from the engagement’s own reporting.
Phase by phase
Phase 1: Unstructured Document Audit
PDF Layout & Scan Quality Analysis
Audited complex PDF invoices, contracts, and scanned forms across various layouts and scan qualities.
- PDF Layout Benchmark
- OCR Quality Metric
- Data Schema
Phase 2: Hybrid OCR & Vision Parser
Layout-Aware Extraction Engine
Engineered a multi-stage parser combining layout-aware Tesseract OCR with GPT-4o vision extraction.
- Vision Extraction API
- Confidence Scoring Pipeline
- Table Parser
Phase 3: Verification & Exception Queue UI
Human Reviewer Interface
Designed a reviewer interface highlighting extracted key-value pairs and flagging low-confidence values for quick manual review.
- Exception Reviewer UI
- Audit Trail Log
- Annotation Tool
Phase 4: Production Rollout & Throughput
High-Volume Processing Pipeline
Processed 100,000+ monthly documents with 99.1% extraction accuracy and zero database sync errors.
- Production Pipeline
- SLA Monitoring Dashboard
- System Clearances
Sixteen seconds from arrival to a posted journal: each step from the OCR and vision passes to validators and sync stored with the record, the balanced entry, and the reversal that record makes possible.
About our collaboration
A cross-functional team of 6 worked on a fixed price, phased basis over 12 weeks, covering Document AI engine, Reviewer UI, Workflow integration. We held a standing mid-week checkpoint and wrote decisions down in place of status meetings. Nothing shipped without a live demo first.
Accuracy was benchmarked against a hand-labeled sample of historical documents before anything was built. That set the target and, more usefully, showed which document types were hopeless and should go to a person by default. The phases were drawn so the reviewer queue shipped before full automation: nothing was ever extracted without somewhere for the doubtful cases to go.
What we'd carry into the next one
- 01
Automated document parsing cut manual data entry processing time by 85%.
The saving was the typing, not the reading: staff still see the exceptions, and the exceptions are where judgment was always needed.
- 02
Hybrid vision OCR achieved 99.1% extraction accuracy across noisy scanned PDFs.
The two passes disagree in exactly the cases that matter, which is why disagreement feeds the confidence score instead of being settled by picking a winner.
- 03
Visual exception reviewer allowed staff to audit low-confidence fields in seconds.
The reviewer is fast because the page and the extraction sit side by side with the field highlighted in place. The previous queue was ignored because it made you hunt.
One invoice, three checks
Held or posted is a sum you can read. The model doesn't get the final say.
Take a faxed freight invoice and an emailed one through what decides their fate: what both passes read, whether the totals recompute, and each field's score against its floor. Switch documents and tabs, or use the arrow keys once a tab is focused.
7 of 8 fields agree · due date does not
| Field | OCR pass | Character confidence | Vision pass | Agree |
|---|---|---|---|---|
| Vendor | BELLWETHER FREIGHT CO. | 0.94 | Bellwether Freight Co. | |
| Invoice number | BF-20931 | 0.91 | BF-20931 | |
| Invoice date | 08/31/2026 | 0.88 | 08/31/2026 | |
| Payment terms | NET 30 | 0.90 | Net 30 | |
| Due date | 09/30/2026 | 0.61 | 09/20/2026 | |
| Subtotal | 7,038.70 | 0.93 | 7,038.70 | |
| Tax | 0.00 | 0.95 | 0.00 | |
| Total | 7,038.70 | 0.90 | 7,038.70 |
Both passes read the same page image, and every value keeps the box it came from. Where they differ is exactly where a faxed copy is hardest to read, so the difference is carried forward as evidence instead of settled by picking a side.
Illustrative mechanism: score = 0.4 × OCR character confidence + 0.6 × agreement, capped at 0.20 when a validator fails; floors of 0.85 for text, 0.92 for identifiers, 0.90 for dates and 0.95 for amounts. The weights and floors are chosen for the example; the documents and amounts are invented.
From a faxed page to a figure you can trace back to it
Python and FastAPI run the passes and the scoring, PostgreSQL holds the output, and reviewers work in a React interface. Nothing syncs without either a score above its threshold or a person who has looked.
- 01 · ArriveInvoices, contracts, statementsScanned PDFs, faxes, images and digital documents from about two hundred vendor layouts enter one pipeline.
- 02 · ReadLayout-aware OCR + vision passTesseract with layout analysis keeps tables as tables; a vision pass reads the same page image alongside it. Every value keeps its bounding box.
- 03 · ScoreComposite confidenceOCR character confidence, cross-pass agreement and the field's validator, against a threshold set per field type.
- 04 · ReviewException queueBelow-threshold fields queue with the page beside them, fully keyboard-operable; corrections are captured as training signal.
- 05 · SyncInternal databaseStructured output written with the source box retained. Zero database sync errors at 100,000+ documents a month.
No confident wrong numbers
Doubt queues, figures trace back, hard cases go to people
Confidence the model can't fake
A score combines OCR character confidence, agreement between the text and vision passes, and the field's own validator. A date that won't parse scores low however sure the model is, and anything under its threshold queues instead of syncing.
Every figure traces to the pixel
Structured output is written to the internal database with the bounding box each value was read from, so a figure disputed in reconciliation goes straight back to the place on the page it came from.
Some documents always get a person
A benchmark on hand-labeled historical documents showed which types were hopeless, and those route to a reviewer by default. The review queue shipped before full automation, so doubtful cases always had somewhere to go.
Still typing figures from scanned invoices and statements into a system by hand? Scope your build in three minutes.
Scope your buildNearby engagements
AI & AutomationA private legal assistant grounded in verified precedents
A private knowledge assistant that searches internal case files and precedents, providing cited answers legal teams can verify in seconds.
Legal & Law Firms · 14 weeks
Product DesignAn onboarding flow that guides trial users to value
A redesigned SaaS trial onboarding experience with progressive checklists, sample data, and inline guidance that turns signups into active subscribers.
Professional Services · 10 weeks
Product DesignA design system that brought speed and consistency to 4 product teams
A token-based design system in Figma and React that eliminated component duplication across 4 product squads and cut the time from design handoff to merged frontend.
Professional Services · 14 weeks
Let's talk
Running a large platform, shaping a first MVP, or getting a product ready for a funding round? Tell us where you are. We'll shape the process around it, and stay with you after launch.
