Engineering a regulated document-AI platform · Chapter 2
LLM operations at billion-token scale
How we run 2.7 billion+ tokens through production with full cost visibility, governed capacity and resilient telemetry.
Facing a similar engineering problem?
Discuss your projectThe engineering problem
In six months the platform has processed 2.7 billion+ model tokens across 275,000+ model calls to Google Gemini models on Vertex AI (measured from usage logs that begin in mid-May; the lifetime total is higher): page OCR, structured extraction, multi-chunk analysis, entity extraction, embeddings and a conversational assistant, in three regions. At that volume, three things decide whether an AI product is viable: knowing exactly what it costs, controlling how fast it spends, and keeping sensitive prompts and outputs governed.
1. Complete usage capture, in every service
Every service that calls a model, embeddings and internal tools included, goes through a shared capture path. Each record carries the service, the call type, the work it belongs to, the data region, and the tokens: input, output and reasoning tokens counted separately (reasoning models spend tokens that never appear in the output).
Captured usage is checked against provider billing, so the cost dashboard reflects what the business actually pays. When a new call type or model is introduced, its pricing is added with it.
Every model call, in every service, goes through one capture path that is checked against provider billing.
Read this diagram as text
- Every service that calls a model, embeddings and internal tools included, goes through a shared capture path.
- Each record carries the service, the call type, the work it belongs to, the data region and the tokens: input, output and reasoning tokens counted separately.
- Captured usage is checked against provider billing, so the cost dashboard reflects what the business actually pays.
2. Governed capacity
- A tokens-per-minute governor in front of model calls, so parallel analysis runs at full speed without exceeding provider quotas.
- A key pool with circuit breakers, so a failing credential or region is taken out of rotation automatically.
- Per-agent timeouts, tuned to each workload (a page of OCR is not a multi-chunk legal analysis).
- Parallel chunked analysis for the largest records, bounded by the governor.
3. Sensitive payloads, governed like health data
Full request and response capture is invaluable for quality work, and it is also some of the most sensitive data in the system. So captured payloads are:
- encrypted at rest, separately from usage metadata;
- viewable only with a dedicated permission, with every view written to the audit trail;
- retained for a limited period, then deleted automatically;
- switchable by administrators, with no deployment needed.
4. Resilient telemetry
At hundreds of thousands of calls, any telemetry writer will eventually see a malformed record. Ours is built so that one bad record doesn't block the rest:
When a batch is rejected because of its data, the writer splits it in half until the malformed record is isolated.
Read this diagram as text
- A batch of 8 rows is rejected because one row is malformed.
- It splits 4 + 4: the clean half is written.
- The half with the malformed row splits 2 + 2, then 1 + 1.
- The malformed row is isolated and logged; every valid row is written.
- If the failure is transient (a network or connection problem), the whole batch is retried instead.
- Usage rows are written in efficient batches.
- If a batch is rejected because of its data, the writer splits it in half, recursively, isolates the single bad record, logs it, and writes everything else.
- If the failure is transient (a network or connection problem), the whole batch is retried.
- Field lengths are validated before insert.
The outcome
- 2.7 billion+ tokens processed with per-service, per-call-type cost visibility.
- Capacity governed per model and provider; failing credentials removed from rotation automatically.
- Sensitive payloads encrypted, permissioned, audited and time-limited.
- Telemetry that isolates a malformed record and writes everything else.
Questions to ask of your own stack
- Does every model call, including embeddings and internal tools, go through one capture path?
- Do you reconcile logged usage against the provider's invoice?
- Is spending rate governed, or does parallelism hit provider limits?
- Is stored prompt and response data encrypted, permission-gated, audited and time-limited?
- If one telemetry record is malformed, do the rest still get written?
Next: Chapter 3, Reliable AI output by design.