We are rebuilding the v2 core of a system that sits between model inference and real-world actions, and I’m looking for criticism from people who have operated production inference infrastructure at scale.
This is not really a benchmark question.
It is a boundary question:
Where should probabilistic inference stop, and where should deterministic execution begin?
Our current high-level path is:
Client / Tool
↓
Canonical Request
↓
Authentication + Purpose
↓
Single-use Authority Grant
↓
Inference Adapter
↓
Normalized Candidate
↓
Deterministic Policy
↓
Protected Action
↓
Journal / Outbox
↓
Receipt Signing
Vertex AI is the current inference backend. A self-hosted path using GKE with vLLM / TensorRT-LLM is under evaluation.
The central invariant is:
model output != authority
The model may propose a candidate action.
It cannot authorize execution.
The deterministic layer evaluates the candidate against the original canonical request, authenticated principal, authority grant, purpose, policy version, expiry, nonce, and replay state.
So the inference adapter is deliberately unable to redefine the scope of the request while normalizing provider output.
1. The trust boundary
I currently think about the system as two domains.
Probabilistic / untrusted
- client-provided task data
- provider response
- model output
- provider metadata
- inference retries
- reasoning / continuation metadata
Deterministic / trusted
- canonical request
- authenticated principal
- authority grant
- policy decision
- replay protection
- execution identity
- execution journal
- protected-action result
- receipt
The adapter validates and normalizes inference output.
It does not grant authority.
The identity chain currently looks roughly like:
RequestID
↓
DecisionID
↓
InferenceAttemptID
↓
CandidateID
↓
ExecutionID
↓
JournalEntryID
↓
ReceiptID
A retry creates a new InferenceAttemptID.
It does not silently create a new decision or consume a new authority grant.
The receipt eventually needs to bind the selected candidate, execution identity, journal entry, authority grant, policy version, and canonical action hash.
Some invariants we are trying to preserve:
model output != authority
no execution without valid authority
no execution without deterministic policy evaluation
no candidate mutation after policy evaluation
no authority replay
no execution identity replay
no receipt claiming an execution
before the outcome is durably journaled
retry != new authority
retry != invisible history
2. Cloud Run and long inference
The control plane currently runs as a stateless Node.js / Fastify service on Cloud Run.
Node can obviously wait on asynchronous inference without blocking the event loop.
The harder question is whether keeping a multi-second inference call attached to the HTTP lifecycle remains a good systems design at high concurrency.
Our current design targets are approximately:
- successful inference p95 < 5 s
- successful inference p99 < 15 s
- up to 10,000 concurrent decisions across providers
Even when the work is asynchronous I/O, an open inference request still consumes concurrency capacity, sockets, memory, upstream capacity, and influences scaling behavior.
So we are considering two models.
Synchronous path
Client
↓
Admission
↓
Inference
↓
Policy
↓
Execution
↓
Response
Decoupled path
Client
↓
Admission
↓
Durable Work Item
↓
DecisionID returned
Then:
Durable Work Item
↓
Inference
↓
Candidate
↓
Policy
↓
Execution
↓
Journal
↓
Result
The second approach could use an outbox plus Cloud Tasks, Pub/Sub, or another durable work mechanism.
It reduces the lifetime of the admission request, but adds state transitions, recovery paths, reconciliation, and different API semantics.
For people who have run multi-second inference at meaningful concurrency:
At what point did you stop keeping inference inside the synchronous HTTP request lifecycle?
For Cloud Run specifically, did concurrency-driven scaling remain predictable under this workload, or did you eventually separate admission from execution?
3. Control plane vs. inference plane
Another architecture we are evaluating is keeping authority-sensitive logic completely outside the GPU infrastructure.
Control plane
- authentication
- authority
- purpose validation
- policy
- replay protection
- execution identity
- journal
- evidence
- reconciliation
Inference plane
GKE
↓
Inference Gateway
↓
vLLM / TensorRT-LLM
↓
GPU Pool
The boundary would effectively become:
Serverless Control Plane
↓
Inference Protocol
↓
Accelerated Inference Plane
The attraction is that the whole application should not need to move onto Kubernetes simply because model serving requires accelerators.
It would also give us independent scaling domains for:
control
inference
journal
signing
reconciliation
But clean abstractions tend to become less clean under saturation and failure.
Has anyone operated serverless admission/control with a dedicated GKE/GPU inference plane?
Where did that boundary eventually become painful?
4. Single-use authority and strongly consistent state
For a few pieces of state, we are targeting RPO = 0:
- authority consumption
- execution identity
- journal records
Single-use authority means the following pattern is unacceptable:
read grant
↓
execute
↓
mark grant as consumed later
Two concurrent callers cannot both observe the same grant as usable.
The semantic operation needs to look closer to:
validate grant
↓
atomically consume grant
↓
bind ExecutionID
↓
commit
Cloud Spanner is currently one of the leading candidates because this state requires a strong transactional consistency model.
The requirement is not “use Spanner because it is Google Cloud.”
The requirement is:
single-use authority
concurrent execution
regional failure
RPO = 0 target
Spanner happens to provide external consistency, which is stronger than single-object linearizability for transactional workloads.
Targets currently under discussion:
- 10,000 state transitions/s sustained
- 100,000 active DecisionIDs
- 10,000 concurrent decisions
- RPO = 0 for authority, execution identity, and journal state
One architecture question I have not closed yet:
Would you keep authority consumption and execution journal state in the same transactional system?
Or deliberately separate them into different failure domains, linked by an immutable ExecutionID?
5. Signing is becoming a separate infrastructure problem
Every successfully protected action eventually needs cryptographic evidence.
The simple implementation is:
EXECUTE
↓
JOURNAL
↓
KMS / HSM SIGN
↓
RESPOND
Semantically clean.
Operationally, it puts a remote asymmetric cryptographic operation directly into the critical path.
The architecture I currently find more defensible is:
EXECUTE
↓
JOURNAL
↓
EXECUTION COMMITTED
And independently:
JOURNAL
↓
SIGNING QUEUE
↓
KMS / HSM
↓
SIGNED
That creates two forms of finality.
Execution finality
The authoritative outcome is durably recorded.
Receipt finality
Cryptographic evidence for that outcome has been produced.
The API could therefore expose states such as:
execution_status = committed
receipt_status = pending
rather than pretending these are the same event.
This matters even more because our current target of 1,000 signatures/s is far above the default regional quota for HSM asymmetric cryptographic requests in Cloud KMS, which is currently documented as 50 QPS per hosting project/region for the relevant quota class. That makes signing throughput a capacity-planning problem, not just an SDK call.
So signing likely needs independent:
- capacity planning
- quota management
- backpressure
- retries
- idempotency
- queue depth monitoring
- key rotation
- failure handling
A receipt should also be idempotent for the same ReceiptID and canonical payload hash.
For people operating high-volume signed logs: where does signing sit relative to your transaction boundary?
Do clients wait for cryptographic finality?
Or do you expose execution finality and receipt finality independently?
6. Model context vs. authority context
Prefix/KV caching creates another boundary problem.
A request may contain all of the following:
stable system context
domain context
task payload
principal
authority grant
nonce
expiry
replay state
I increasingly think these should not occupy the same semantic layer.
Model context
- system instructions
- domain context
- task payload
Execution context
- canonical request
- authenticated principal
- authority grant
- purpose
- policy version
- nonce
- expiry
- replay state
The model answers:
What action appears appropriate?
The deterministic system answers:
Is this principal authorized to execute exactly this action right now?
Those are deliberately different questions.
This also potentially improves prefix/KV-cache reuse.
The key rule would be:
cache reuse == computation reuse
cache reuse != authority reuse
A cache hit must never imply permission.
For people doing prefix-aware routing or KV-cache reuse in multi-tenant systems:
Have you deliberately kept authorization state outside the model-context identity?
What problems did you encounter?
7. Retries are not implementation details
A transient inference failure can create this situation:
Attempt #1
↓
Candidate A generated
↓
response lost
Then:
Attempt #2
↓
Candidate B generated
To ordinary infrastructure this looks like a retry.
To an execution system there may now be two different proposed actions.
So we treat attempts as first-class objects:
DecisionID
├─ InferenceAttemptID #1
│ └─ Candidate A
│
├─ InferenceAttemptID #2
│ └─ Candidate B
│
└─ Selected CandidateID
Execution references exactly one immutable candidate.
The rules become:
retry != new DecisionID
retry != new authority
retry != invisible history
Our current automatic retry budget is at most two attempts per decision.
Ambiguous outcomes remain attributable instead of disappearing behind retry logic.
I would be very interested in how others represent retry identity when model inference itself is non-deterministic.
8. Provider metadata should stop at the adapter boundary
Modern model APIs return much more than text or tool calls.
Depending on the provider, a response may contain:
- reasoning state
- continuation state
- provider-specific signatures
- internal identifiers
- safety metadata
- finish states
- protocol-specific fields
Our boundary currently looks like:
Provider Response
↓
Validate
↓
Normalize
↓
Normalized Candidate
Provider-specific internal state terminates inside the adapter unless an explicit protocol requirement says otherwise.
It should not accidentally become:
- policy input
- execution identity
- receipt content
- signed evidence
- audit assertions
But there is an opposite danger.
Over-normalization can destroy useful semantics.
Our provider-neutral representation therefore still needs to distinguish things like:
success
refusal
truncation
tool call
malformed tool call
invalid output
provider rejection
transport failure
unknown completion state
For people who have built multi-provider inference abstractions:
What did you normalize too aggressively?
What provider-specific field looked irrelevant until production proved otherwise?
9. Runtime state, durable state, and cryptographic finality are different
Another rule we are trying to preserve is:
runtime observation
!=
durable state
!=
cryptographic finality
The conceptual lifecycle is:
PREPARED
↓
AUTHORIZED
↓
EXECUTED
↓
JOURNALED
↓
SIGNED
↓
RECONCILED
But those transitions deliberately mean different things.
EXECUTED
The downstream operation appears to have occurred.
JOURNALED
Its authoritative result has been durably committed.
SIGNED
Cryptographic evidence has been committed for that result.
RECONCILED
The recorded result has been matched against the authoritative downstream state.
An ambiguous execution should be able to become:
EXECUTED
↓
OUTCOME_UNKNOWN
↓
RECONCILED
rather than triggering a blind retry that might duplicate an external action.
Current design targets
These are targets we want to test against, not production benchmark claims.
Latency
- authorization/admission p95 < 100 ms
- authorization/admission p99 < 250 ms
- inference gateway overhead p95 < 20 ms
- non-inference overhead p99 < 500 ms
- inference including retries p95 < 5 s
- inference including retries p99 < 15 s
- adapter normalization p95 < 10 ms
- asynchronous journal acknowledgement p95 < 250 ms
- signed receipt after journaling p95 < 5 s
Scale
- control-plane admission: 1,000 requests/s
- state transitions: 10,000/s sustained
- concurrent decisions: 10,000
- active DecisionIDs: 100,000
- inference: 100 requests/s sustained
- inference burst: 1,000 requests/s
- signing design target: 1,000/s
- pending reconciliation: 100,000
- pending unsigned receipts: 10,000
Durability
- RPO = 0 target for authority consumption
- RPO = 0 target for execution identity
- RPO = 0 target for journal records
- control-plane RTO < 5 minutes
The actual question
A lot of excellent infrastructure work is happening around:
- continuous batching
- prefix-aware routing
- KV-cache management
- disaggregated prefill/decode
- speculative decoding
- GPU scheduling
- LoRA-aware routing
But once model output can trigger an external action, there seems to be another systems boundary worth treating explicitly:
PROBABILISTIC INFERENCE
↓
HARD BOUNDARY
↓
AUTHORITY
↓
POLICY
↓
EXECUTION
↓
JOURNAL
↓
EVIDENCE
Would you keep this boundary this hard?
I’m particularly interested in production failures rather than ideal architectures:
- At what concurrency did synchronous inference become the wrong HTTP architecture?
- Did serverless control + dedicated GPU inference remain a clean boundary?
- How did you implement single-use authority under regional failure?
- Where did you put KMS/HSM signing relative to execution finality?
- Did signing eventually become its own scaling domain?
- What did prefix/KV caching accidentally couple together?
- How did you represent retries for non-deterministic inference?
- What did your provider abstraction throw away too early?
- Which p95/p99 threshold actually forced an architectural change?
- Which consistency guarantee looked necessary on paper but was not worth its operational cost?
Architecture criticism is very welcome — especially incident patterns, numbers, failure thresholds, and mistakes you would not repeat.