Document Processing at Scale
Building intelligent document parsing and classification for a legal AI startup. From prototype to 5M daily transactions in 18 months.
The Challenge
The team had a working prototype that classified documents with 94% accuracy. But the prototype was not production code. It had no error handling, no logging, no way to catch and re-process failed documents. When they tried to scale to 100K documents per day, the entire pipeline stalled.
The real problem was not accuracy. It was architecture. A machine learning pipeline that works on your laptop breaks in three ways in production: (1) data drift — models trained on Tuesday don't work on Friday because the real world changed; (2) failure isolation, if one step fails, does the whole pipeline fail, or just that document?; (3) observability — when something breaks, can you see it before customers do?
They had also made a bet on in-house ML infrastructure (not a third-party service like AWS Textract). That meant owning the entire stack: model training, inference servers, result caching, API rate limiting, error recovery. Scale from 100K to 5M documents per day is not a 50x increase in compute. It is a re-architecture.
Our Approach
We started with a 2-week architecture audit (the 48-hour audit is our standard entry point, but this project needed deeper scoping). We mapped the entire ML pipeline: data ingestion, preprocessing, inference, post-processing, storage, and API layer. We identified the bottleneck (inference was single-threaded) and the failure points (no retry logic, no rollback, no circuit breaker for downstream services).
Then we designed for the next 10x. Not the next 2x. If they were going to own this infrastructure, it had to be designed for 50M documents per day, with graceful degradation, with observability that would catch problems before customers did.
The Architecture Pillars
We rebuilt the pipeline around four principles: (1) Isolation: Each stage (ingestion, preprocessing, inference, post-processing) runs in its own worker pool. One failure does not cascade; (2) Observability: Every document has a trace ID. We log every state transition. We ship metrics to monitoring. No mystery failures; (3) Resilience: Exponential backoff with jitter on retries. Dead-letter queues for documents that fail after 3 attempts. Circuit breaker on downstream APIs; (4) Scaling: Horizontal. Add workers, not bigger workers. Use managed queues, managed caches, managed databases. Allows for independent scaling of each stage.
What We Built
A distributed document processing pipeline on Kubernetes. The architecture uses message queues (RabbitMQ) to decouple stages, so each stage can scale independently. We containerized the ML inference engine (TensorFlow Serving) so it runs as a managed service, not a custom Python process. We added a Redis cache layer in front of inference to catch re-processing of identical documents.
For observability, we wired in Datadog. Every document fires logs and traces. We built a dashboard that shows documents in flight, error rates per stage, model accuracy on recent batches. We added a dead-letter dashboard so the support team could re-process failed documents without engineering involvement.
We also built a versioning system for models. New model versions don't replace old ones. They run in parallel, with a small percentage of traffic. We compare accuracy and latency. If the new model wins, we shift more traffic. If it loses, we roll back instantly. No multi-hour model rollouts, no data corruption.
Results
"We went from 'will this blow up at 1M documents per day' to 'we can grow 10x and nothing breaks.' That freed us to focus on product, not infrastructure firefighting."
— CEO, Client AI/ML Startup
Ready to Build?
Whether you need a 48-hour architecture audit or a full 6-month rebuild, we start with thinking, not code.