Engineering a regulated document-AI platform at billion-token scale

The client needed an AI platform that reads complete medical records (scanned PDFs running to thousands of pages, Word files, photographs, DICOM imaging studies, video and audio) and turns them into a structured, cited analysis that lawyers and clinicians can rely on. A reviewer must be able to question any case and get answers cited to the exact source page.

The constraints set the bar:

• Scale and variety. Records range from a few pages to more than 10,000, across PDFs, Word files, images, DICOM studies, video and audio.
• Regulation. Health data must stay in its jurisdiction, every access must be auditable, and the platform must stand up to independent security testing and sector assurance reviews.
• Trust. Outputs inform legal and clinical decisions, so every result must be provably complete and traceable to its source.
• Time. The client needed to be in production in weeks, then keep shipping while real customers used it.

Webexa owned the engagement end to end: product architecture, AI engineering, cloud infrastructure, security and compliance engineering, delivery, and production operations.

We started with decisions, not code:

• Recorded architecture decisions: significant choices documented with context, the options weighed, the evidence, and the trade-off accepted.
• Evidence over preference. Where the right technology was unclear, we built a gold-set evaluation on real data and let the numbers decide (see Choosing technology with evidence).
• Compliance as architecture. Auditability and encryption are part of the core platform, and data residency is enforced at the system boundary: each new region is a complete, isolated stack.
• Leverage where it is safe. A commercial SaaS foundation for commodity concerns (auth, teams, billing, admin), so engineering time went into the hard parts: the document pipeline, the AI layer, and the regional architecture.
• Configuration as data. Plans, prices, prompts, feature flags, email templates and translations are database-backed and admin-editable, so the product can change daily without a deployment.

The result: production in about nine weeks, followed by continuous delivery of new capability.

A document pipeline engineered for 10,000-page records. Every document is split into pages, and each page is processed as an independent, retryable, resumable task. One upload can fan out into thousands of parallel tasks. The pipeline accounts for every page: it saves the expected page count up front, reconciles it against completed pages, and holds any result that isn't complete. Page subsetting keeps intermediate files on image-heavy pages ~1,000x smaller than a naive split, and workers scale on queue depth, because model-bound work is IO-bound and CPU is the wrong signal. → Chapter 1: Engineering document AI for 10,000-page records

An AI layer built for production, not demos. Google Gemini models on Vertex AI read every page as an image. Extraction and analysis run in parallel chunks under a tokens-per-minute governor, behind a key pool with circuit breakers and per-agent timeouts. Generation is constrained: facts are computed in code, the model writes within schemas, and structural violations go back to the model to rewrite. Every model call in every service is metered, and stored payloads are encrypted, permission-gated, audited and time-limited. Over 2.7 billion tokens have run through it. → Chapter 2: LLM operations at billion-token scale · Chapter 3: Reliable AI output by design

Case knowledge with cited answers. Per-case entity extraction feeds a three-node, highly available graph database with native vector search, holding ~450,000 nodes. A streaming assistant answers questions with page-level citations. On the largest records we parallelised entity extraction, taking it from ~14 hours to ~15 minutes (~56x). → Chapter 5: Choosing technology with evidence

Three regions, each a full, highly available stack. An edge router sends every user to their data's region. Only global configuration is mirrored between regions; health data never leaves its own. Each region runs on multi-AZ managed databases behind a connection proxy with per-connection IAM authentication (credential rotation is removed from the client path), redundant networking, replicated schedulers, and container services with minimum healthy capacity. → Chapter 7: Data residency as an engineering boundary

Zero-downtime delivery. Every service has a health-check contract (it fails only on its own hard dependencies), hard stability gates, and automatic rollback in CI. Under test, a full replacement of the web fleet served 118 consecutive probes with zero errors, and killing a live assistant task mid-traffic kept 120 of 120 health probes green. → Chapter 6: Deployments and monitoring that prove success

• Data residency by design: three regional stacks, regional storage and keys, an edge router, and mirroring limited to global configuration.
• A tamper-evident audit trail: database-level protection against edits and deletes, hash-chained entries, and compliance-mode write-once archival for seven years. 368,000+ audit events recorded.
• Encryption everywhere: customer-managed keys for storage, databases and secrets; TLS on every external and database connection; encrypted model-payload capture with permissioned, audited access.
• Independent assurance: an independent penetration test with every finding closed, a security-framework readiness register, and records of processing for data-protection regulators.
• Least privilege by default: short-lived OIDC credentials for deployments, IAM-authenticated database connections, and a web application firewall at every regional edge.

→ Chapter 8: Making compliance requirements testable

Our engineers build alongside frontier coding agents. The engineers own every decision; the agents speed up the work inside a process built for regulated production: