Economics, architecture,
and the optimization loop
How token cost is reduced, how the system is deployed, how the optimization loop runs, and how cost is governed.
Route, serve, train, and govern
solid yellow = serving / request flow
grey dashed = training / data flow
Functional Requirements
Training loop
Which agents are worth training, the run itself, the on-policy rollout cycle, and the ladder that proves a candidate.
Akka orchestration · hours to daysServing loop
The hot path, how a candidate earns real traffic, and the drift watch that sends a decayed agent back to T1.
Governance
The one orchestrated pass that admits and captures every request, the standing requirements, and which controls need a person.
Gateway + policyScorecards
What the loop costs and whether quality held, per audience — and the decisions it is waiting on a person to make.
Read sideTarget & Admit
Akka orchestrationWhich agents are worth training. A policy filter, two checks the agent must pass, and two scores that rank whatever survives — returning a named reason for every verdict.
Flowpolicy filter → four scheduled scorers → one verdict
solid = invocation · grey dashed = read · red dashed = failure path · green = must pass, a zero disqualifies the agent · yellow = affects rank only
Trigger
A schedule, or a hand-back from S3 when a promoted model decays. Each run scores one agent across a window of its recent traffic, so there is one verdict per agent rather than one per request.
State & durability
Each scorer saves its result against the agent it scored. The final verdict is a live view over those four saved scores, so whenever any scorer saves a new one, the verdict recomputes. Nothing schedules the verdict itself — it moves whenever a scorer moves. A crash part-way through resumes at the last completed step.
Failure & compensation
A failed check renders KEEP_FRONTIER with the reason named — "ungradeable, no deterministic grader for this task shape", "only 31 distinct examples, below the minimum", "policy veto: regulated task". The verdict and its reason are written to the record.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| Narrowness | POST /v1/embeddings | vLLM serving an embedding model inside your VPC | Fireworks embeddings endpoint | vectors it can cluster, to measure how much the wording varies |
| Gradeability | POST /v1/chat/completions | vLLM serving the judge model inside your VPC | Fireworks, or the frontier TEACHER tier | a yes/no on whether a checkable rubric can be built for an unseen task shape |
| Volume sufficiency | POST /v1/embeddings | the same vLLM endpoint, results cached | the same Fireworks endpoint | which requests are near-duplicates, so repeats collapse to a count of distinct examples |
| Token economics | — none | a tokenizer library and a price book, both in-process | token counts and spend per task shape | |
Both postures speak the same OpenAI-compatible API, so only the base URL differs between them — this workflow's code is identical either way.
Components to buildevery component T1 needs, and the scaffolding each one depends on
That code was written for the sales assessment tool — a one-shot run over a customer's exported logs, ranking which use cases and how many tokens are eligible for training. T1 is the same algorithm running continuously. The scoring logic ports directly; the orchestration does not.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Workflows4durable multi-step processes — each resumes at its last completed step | |||
| NarrownessScoringWorkflow | WF | Reads the agent's recent window, embeds a bounded sample, clusters it, and writes the A1 sub-score | Trace-window reader · embedding client · reservoir sampler |
| GradeabilityScoringWorkflow | WF | Sorts the agent's traffic into grader tiers and writes the A2 sub-score | Grader-definition registry |
| VolumeScoringWorkflow | WF | Collapses near-duplicates to a distinct-example count, measures accrual, and writes the A3 sub-score | Near-duplicate collapse over the window |
| EconomicsScoringWorkflow | WF | Decomposes spend across system, context, reasoning, output and retry, and writes the A4 sub-score | Price book · tokenizer |
| Timed Actions3schedule only — each starts a run and does no work itself | |||
| NarrownessTimedAction | TMR | Nightly. Starts one narrowness run per eligible agent — it does no scoring of its own | Self-rescheduling timer, bootstrapped at startup |
| VolumeTimedAction | TMR | Hourly. Starts one volume run per eligible agent | Self-rescheduling timer |
| EconomicsTimedAction | TMR | Every tick — cadence undecided, see below. Starts one economics run per eligible agent | Self-rescheduling timer |
| Consumers2react to a stream of changes | |||
| GraderDefinitionConsumer | CON | Reacts to a grader-definition change and starts a gradeability run for each agent it affects | Grader-definition registry to subscribe to |
| NewCandidateConsumer | CON | A8. Streams admission changes and emits who newly qualifies | Change feed off the reducer view |
| Entities8durable state, one instance per agent | |||
| NarrownessScoreEntity | ENT | Holds the A1 sub-score for one agent | — |
| GradeabilityScoreEntity | ENT | Holds the A2 sub-score for one agent | — |
| VolumeScoreEntity | ENT | Holds the A3 sub-score for one agent | — |
| EconomicsScoreEntity | ENT | Holds the A4 sub-score for one agent | — |
| AdmissionEntity | ENT | Per-agent verdict — TARGET or KEEP_FRONTIER, its named reason, and the A7 estimate | — |
| TrainingStateEntity | ENT | Lifecycle: none → admitted → training → in-ladder → promoted or retired. Control plane only; the serving path never reads it | — |
| EligibilityPolicyEntity | ENT | A9 pre-filter vetoes and late modulators, per agent | Policy source of record |
| InteractionEntity | ENT | One sampled interaction — the trace substrate every scorer reads | — |
| Views2queries that recompute when their sources change | |||
| AdmissionReducerView | VIEW | A6 composite over the four sub-score entities. Any write re-runs it; nothing schedules it | — |
| InteractionsByAgentView | VIEW | Groups interactions by agent so a scorer can read one agent's window | — |
Four sub-score entities, not one. The assessment tool holds all four scores in a single entity and gets away with it because one process computes them serially. Under T1's independent cadences, four writers against one entity is last-writer-wins — the hourly volume write would clobber the nightly narrowness write. Splitting them is what makes the reducer-as-a-view design work at all.
The trigger is not the work. A timed action has no journal and no steps — on failure it re-runs from the top, indefinitely by default. So each trigger does exactly one thing: start the run. The run is a workflow, which is what makes "resumes at the last completed step" above true rather than aspirational. There is no workflow that orchestrates all four; each scorer runs on its own schedule, and the view recomputes whenever any of them writes.
How often is "every tick"? Engineers must decide this, and it is the biggest open question on this slide. It sets how many workflow instances exist at once, so it is a cost decision before it is a correctness one. The instinct to batch is probably wrong: with a large fleet under evaluation, many small incremental runs are likely better than few large ones — a long cadence lets a backlog of un-scored agents build up, and the deeper that backlog, the staler every verdict downstream of it. Start frequent and incremental, then measure.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Template extraction + structural entropy | A1 | Measures how much the request shape varies |
| Tool-path entropy | A1 | Measures how much the call sequence varies |
| Output-space classification | A1 | A fixed label, JSON or SQL scores narrow; open prose scores wide |
| Semantic clustering over embeddings | A1 | Turns vectors into a dispersion number — streaming, over a bounded sample |
| Near-duplicate collapse | A3 | Repeats must not count as distinct training examples |
| Grader-tier detection | A2 | Sorts traffic into deterministic, model-judgeable, human-only, or ungradeable |
| Token accounting + waste decomposition | A4 | Splits spend across system, context, reasoning, output and retry |
| Composite gating | A6 | Two must-pass checks multiply; two weights rank whatever survives |
| Estimate waterfall + feasibility | A7 | Magnitude and confidence, computed separately and never multiplied together |
| Eligibility policy evaluation | A9 | Applies vetoes and modulators, always carrying a reason |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why T1 needs it | Constraint |
|---|---|---|---|
| Text embedding | Model inference | A1 semantic entropy and A3 near-duplicate collapse both need vectors | Must run in-VPC. Logs may not leave the customer boundary — that rules out hosted embedding APIs |
| Tokenizer | Library, in-process | A4 counts tokens whenever the source log doesn't carry them | Must match the frontier model's encoding, or the spend number is wrong |
| Price book | Reference data, not an API | A4 turns token counts into money | Needs a refresh path — prices move |
| Judge / chat inference | Model inference | A2 only, and only in V2 — asks whether a checkable rubric could be built for a task shape it hasn't seen | Optional. V1 ships without it: a task with no deterministic grader simply fails the gradeability check |
| Trace store | Data substrate | Every scorer reads past requests and replies, grouped by agent | Windowed and sampled reads, never full scans |
| Grader-definition registry | Data substrate | A2 reads which graders exist; its change events are what trigger the scorer | Not built — and nothing today subscribes to it |
| Governance / policy source | Data substrate | A9 reads regulated status, training consent, and data residency | Customer-owned. A wrong answer here is a compliance failure |
Sanitization — and where it must not be applied. Akka's sanitizer masks matched patterns (email, phone, card, IBAN, IP, plus custom rules) and can be called from any component, not only an agent. Enable it on logs, where it costs nothing, and apply it to the A2 judge call if that judge is not served in-VPC — the one place in T1 where text leaves the boundary.
Do not sanitize before tokenizing or embedding. Masking rewrites the text, and this workflow exists to measure text: a masked span changes the token count that A4 prices, shifts the vectors A1 and A3 cluster, and can alter the extracted template. It would corrupt the scores silently, since nothing would fail. Measure first, sanitize after.
How the embedding capability is wired today — a local daemon if one is running, else an embedding model bundled in-process, else a deterministic fallback that reports itself as degraded. All three keep logs inside the boundary. Production resolves the same capability against the vLLM-on-KServe stack the product already deploys: same interface, different base URL.
Training Run
Akka orchestrationDays long and resumable at every stage. The orchestrator holds the run, so a GPU failure costs a step. The two deployment postures both train — they simply do it through different interfaces.
Flowone trigger · sequential stages · posture branch
solid = invocation · grey dashed = read · red dashed = failure path · green = the run passed its own check
Trigger
T1 admits an agent, and TrainingState(agent) shows no run already in flight. That state is a control-plane entity — the serving path never reads it.
State & durability
Long-running, with durability at every checkpoint. Each stage records its inputs by content hash, so a completed run is reproducible and a failed one resumes at its last checkpoint. TrainingState moves admitted → training → in-ladder, or to retired.
Failure & compensation
Infrastructure failure resumes from the last checkpoint. A run that fails its exit check is retired and never reaches the ladder. The incumbent keeps serving throughout — the request plane is untouched until T4 promotes.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| Dataset build | POST /v1/chat/completions | the frontier TEACHER tier, reached through the gateway like any other call | teacher trajectories to learn from | |
| Training job | — not an inference call | launch a verl job against capacity you reserved | post a fine-tuning job; capacity is managed for you | a job handle to poll, then an adapter |
Inference is uniform across postures, so the teacher call is identical on both. Job submission is not — these are two different services, and the own-GPU path is the only one where you reserve capacity.
Components to buildevery component this workflow needs, and the scaffolding each one depends on
One long-running run per admitted agent. The substrate branch is real — own-GPU and managed tuning are not one API behind a flag, so the training step forks and everything around it stays common.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Workflows2durable multi-step processes — each resumes at its last completed step | |||
| TrainingRunWorkflow | WF | Dataset build → augmentation → train → emit checkpoint → exit check. Long-running; resumes from the last completed stage | Content-hash manifest per stage · substrate branch |
| TeacherHarvestWorkflow | WF | Replays logged tasks through the frontier tier to harvest teacher trajectories for the dataset | Gateway route to the TEACHER tier |
| Consumers1react to a stream of changes | |||
| AdmissionConsumer | CON | Reacts to a TARGET verdict from T1 and starts a run — but only when training state shows none in flight | Admission change feed |
| Entities3durable state, one instance per key | |||
| TrainingStateEntity | ENT | Shared with T1. The interlock that stops a second run starting while one is in flight | — |
| CheckpointRegistryEntity | ENT | Checkpoints produced for an agent, addressed by content hash | Artifact store |
| DatasetManifestEntity | ENT | What went into a run — sources, filters, split — by content hash, so a result is reproducible | — |
| Views1queries that recompute when their sources change | |||
| RunsByAgentView | VIEW | Run history per agent; what the in-flight interlock and the dashboards read | — |
The substrate branch is not a flag. Own-GPU launches a training job against capacity you schedule and writes a checkpoint to your store. Managed tuning posts a job and hands back an adapter, and has no capacity reservation at all. Only the training step differs; dataset build, augmentation and the exit check are common to both.
The exit check is not T4's first rung. This asks whether the run produced a usable checkpoint — converged, no collapse, better on the trainer's own validation split. Whether it beats the incumbent is a different question, asked by T4 on different data. Conflating them is how a bad candidate reaches real traffic.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Dataset assembly from traces | Selection, filtering and the train/validation split over captured production traffic | |
| Synthetic augmentation | Expands thin task shapes so a run has enough coverage to learn from | |
| Exit check | Converged, no collapse, improved on the run's own validation split | |
| Content-hash stage manifest | Makes a run reproducible and lets a failed run resume rather than restart |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why this workflow needs it | Constraint |
|---|---|---|---|
| Frontier teacher inference | Model inference | Harvests teacher trajectories to learn from | Routed through the gateway so training traffic is governed like production traffic |
| RL training runtime | Compute job | Runs the training itself on the own-GPU posture | You schedule the capacity; the split between training and serving is yours to make |
| Managed fine-tuning service | Managed job | The same outcome on the managed posture | No capacity call exists here — the provider owns scheduling |
| GPU capacity scheduling | Infrastructure | Reserves the share of the cluster a run may use | Own-GPU only. This has no counterpart on the managed path |
| Artifact store | Data substrate | Holds checkpoints and adapters | Content-addressed, so a checkpoint is identified by what it contains |
Rollout Loop
Akka orchestrationLearning from the real harness. The agent works in a sandbox at a pinned commit, the environment marks what it did, and the result feeds the next training step.
Flowclosed cycle · repeats until the trainer stops asking
solid = invocation · grey dashed = result returning · red = the boundary the agent cannot cross
Trigger
RolloutRequested from the trainer in T2. Every cycle is one request, so a training run may drive thousands of them.
State & durability
Event-driven between the trainer and the orchestrator. Sandboxes are temporary; their lifecycle is durable, so one that is orphaned gets reclaimed rather than left running.
Failure & compensation
Anything reversible runs for real inside the sandbox. Anything irreversible is faked. That is set by how the sandbox is built rather than by configuration, so it holds even when the model tries something unexpected. A rollout that hangs is stopped and its attempt discarded.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| Act | POST /v1/chat/completions | vLLM holding the staged checkpoint, reached through the gateway | Fireworks holding the staged checkpoint, same path | the agent's next move in the harness |
| Mark the work | POST /v1/chat/completions | vLLM serving the judge model in your VPC | Fireworks, or the frontier TEACHER tier | a score where no test can decide it |
| Mark the work | — not an inference call | the commit's own test suite, executed inside the sandbox | pass or fail, with no model involved | |
Marking the work is two different things wearing one name. Where the commit has tests, they decide it and nothing is asked of a model. A judge is called only for work no test can settle.
Components to buildevery component this workflow needs, and the scaffolding each one depends on
One rollout is one episode: stage the checkpoint, provision a sandbox at a known commit, act, grade, resync. This is the only workflow that runs a model against a live workspace, which is why the containment sits in the authorization layer rather than in the prompt.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Workflows1durable multi-step processes — each resumes at its last completed step | |||
| RolloutWorkflow | WF | One episode — stage → provision → act → grade → resync. Owns the sandbox lifecycle, so an orphan is always reclaimed | Sandbox runtime · repo at a pinned commit |
| Agents1model calls with tools, guardrails and session memory | |||
| RolloutAgent | AGT | Drives the task inside the sandbox against the pinned checkpoint | Vendor harness parity · scoped tool set |
| Consumers1react to a stream of changes | |||
| RolloutRequestConsumer | CON | Reacts to a rollout request from the training backend and starts an episode | Backend event contract |
| Entities2durable state, one instance per key | |||
| SandboxEntity | ENT | Durable lifecycle of an ephemeral sandbox — what makes orphan reclamation possible | Sandbox runtime |
| TrajectoryEntity | ENT | The captured episode: model I/O and workspace state, without which a verifiable reward cannot be recomputed | — |
| Views1queries that recompute when their sources change | |||
| RolloutsByRunView | VIEW | Episodes per training run — progress, sample count, discards | — |
Containment is authorization, not prompting. The rollout runs as a scoped, read-and-sandbox-only projection of the real persona's identity and tool access. Reversible in-workspace tools run for real inside an ephemeral clone; irreversible or external tools are mocked. Enforced at the authorization layer — never by trusting the model. Akka's guardrails do not implement this: they inspect text at the model boundary, and this is an access-control question.
Train must equal serve. If the rollout harness differs from the harness at serving, the reward optimizes an environment that does not exist. Prefer driving the real vendor harness in the loop, and snapshot workspace state rather than model I/O alone.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Tool taxonomy enforcement | Decides what runs for real in the sandbox and what is mocked | |
| Verifiable reward computation | Test execution inside the sandbox — the part of the reward that is not a model opinion | |
| Trajectory capture | Records model I/O together with workspace state at a pinned commit | |
| Sandbox lifecycle and reclamation | Ephemeral by design, durable in bookkeeping, so nothing leaks |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why this workflow needs it | Constraint |
|---|---|---|---|
| Pinned-checkpoint inference | Model inference | The rollout traffic itself, against the staged checkpoint | Reached through the gateway, so it is metered and recorded like production traffic |
| LLM judge | Model inference | Scores only the subjective portion of the reward | The verifiable portion is test execution and is not a model call |
| Sandbox runtime | Infrastructure | Provides the ephemeral, isolated workspace an episode acts in | Must be disposable and reclaimable; an orphan costs money silently |
| Source host | Data substrate | Supplies the repository at a pinned commit | The commit is part of the trajectory — a moving branch makes rewards irreproducible |
| Identity and access control | Governance | Projects a scoped, least-privilege persona for the episode | This is the containment boundary. Nothing else prevents a real write |
| Vendor agent harness | Runtime | The loop the agent actually runs in | Must match serving, or the reward optimizes the wrong environment |
Prove & Promote
Akka orchestrationFour steps, each risking a little more than the last, and any of them can send the candidate back. Two of the four are carried out by S2 inside the gateway; this workflow keeps the score.
Flowfour steps · rising risk · one shared way back
solid = invocation · grey dashed = read · red dashed = the way back · green = no user is exposed at this step
Trigger
CheckpointReady, emitted by T2 when a run passes its own check. The ladder starts on that event rather than by polling for new checkpoints.
State & durability
Each step records its verdict and the evidence behind it. Promotion is a single request to the gateway, which applies it in one move — that is what makes undoing it the same size of change. The gateway is the only writer, so every switch has one order and one record.
Failure & compensation
Any step failing compensates identically — back to the model in service, keep answering, set the candidate aside, raise an alert. At the canary step S2 reverts on its own without waiting for this workflow, then reports the breach so the verdict and its evidence are still recorded here.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| Compare it offline | POST /v1/chat/completions | vLLM holding the candidate, with no traffic pointed at it | Fireworks holding the candidate, same | answers to score against the model in service |
| Compare it offline | POST /v1/chat/completions | vLLM serving the judge model in your VPC | Fireworks, or the frontier TEACHER tier | a quality score for each answer |
| Shadow · Canary | — delegated to S2 | this workflow sets the share and reads the outcome; the gateway does the mirroring and splitting | a verdict per step | |
| Promote | — not an inference call | a request to the gateway, which is the only thing allowed to change what serves | confirmation of what changed | |
Components to buildevery component this workflow needs, and the scaffolding each one depends on
Four rungs, each risking a little more than the last, and any of them can send the candidate back. Two rungs are carried out by S2 inside the gateway; this workflow owns the saga and keeps the score.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Workflows2durable multi-step processes — each resumes at its last completed step | |||
| PromotionWorkflow | WF | The four-rung saga. Sequences the rungs, records each verdict with its evidence, and emits the promote intent at the end | Rung acceptance thresholds · S2 split control |
| OfflineEvalWorkflow | WF | Rung 1 — loads the candidate and scores it over held-out production traffic. No traffic at risk | Held-out eval set · adapter load |
| Consumers2react to a stream of changes | |||
| CheckpointReadyConsumer | CON | Reacts to a checkpoint from T2 and opens a promotion saga | T2 event contract |
| CanaryBreachConsumer | CON | Receives a breach from S2, records the rung verdict, and quarantines the candidate | S2 breach event |
| Entities3durable state, one instance per key | |||
| CandidateEntity | ENT | The candidate under promotion — rung history, evidence, and current hold state | — |
| PromotionEvidenceEntity | ENT | The standard the candidate actually proved at promotion. Retained, because S3 measures drift against it later | — |
| TrainingStateEntity | ENT | Shared with T1 and T2. Moves to promoted or retired as the saga ends | — |
| Views2queries that recompute when their sources change | |||
| CandidatesByAgentView | VIEW | Candidates and their rung position per agent | — |
| DecisionQueueView | VIEW | Candidates holding at a rung because a person must decide — the queue D2 renders | — |
Promotion is one intent, which is what makes rollback cheap. This workflow does not write the routing table; it asks. The gateway applies the change atomically, so undoing it is the same size of change — no drain, no redeploy. A rollback that costs more than a promotion does not get used in an emergency.
Some holds are deliberate. A candidate stops at its last passed rung when the agent is high blast radius, or when cost improves while quality is only level. A person decides that; no threshold does. The candidate keeps its evidence while it waits.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Rung sequencing and saga state | Advances or retires a candidate, and compensates when a rung fails | |
| Acceptance-bar comparison | Quality against the incumbent, and token reduction against the target | |
| Paired-score aggregation | Turns per-request comparisons into a rung verdict | |
| Quarantine | Sets a failed candidate aside with its evidence rather than deleting it |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why this workflow needs it | Constraint |
|---|---|---|---|
| Adapter load | Model serving | Stages the candidate so the offline batch can run against it | Serving-tier operation; the posture decides how it is expressed |
| Batch inference | Model inference | Scores the candidate over the held-out set | Offline — no production traffic is exposed at this rung |
| LLM judge | Model inference | Supplies the quality half of the acceptance bar | Must be the same judges S3 later measures drift with, or the standard moves |
| Routing intent | Control plane | Asks the gateway to switch what serves | The gateway is the sole writer; this workflow never writes routing itself |
Route & Serve
GatewayThe hot path. Every request from a coding tool arrives here and leaves with an answer; a lookup table decides which model produced it.
Flowone pass, in order · one way out to the frontier
solid = invocation · grey dashed = read · red dashed = the way out to the frontier
Trigger
A request arrives from a coding tool, in the same format those tools already speak. Pointing them here is an address change and nothing else.
State & durability
Nothing per request is kept — this path is measured in milliseconds. The lasting object is the lookup table: which model and version answers for each agent. The gateway is its only writer; T4, S3, and human approvals ask, and the gateway makes the change.
Failure & compensation
A timeout, an error, or an answer below the confidence policy goes out to the frontier instead. Every request comes back with an answer. How often that happens is itself measured — a rising rate re-opens S3.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| Answer | POST /v1/chat/completions | vLLM serving the base model with the agent's add-on loaded | Fireworks serving the same pairing | the answer, plus token counts and timing |
| Out to the frontier | POST /v1/chat/completions | the frontier provider's own API, reached through the gateway so it is recorded like anything else | an answer from the strongest available model | |
| Look up the model | — not an inference call | the gateway's own lookup table, held in memory on the request path | which model and version to use | |
Components to buildevery component this workflow needs, and the scaffolding each one depends on
The request path. Nothing here is durable per request — this path is measured in milliseconds. The lasting object is the routing table, and the gateway is the only thing allowed to write it.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Endpoints1the inline request path — no durable per-request state | |||
| GatewayEndpoint | EP | OpenAI-compatible ingress. Classifies, selects the tier, binds the adapter, serves, and returns usage. The only inline hop | Serving tier · routing table held in memory |
| Consumers1react to a stream of changes | |||
| RoutingIntentConsumer | CON | Applies intents from T4, S3 and human approvals, then emits what it actually did | Intent event contract |
| Entities2durable state, one instance per key | |||
| RoutingTableEntity | ENT | Agent → tier → adapter version. One writer, one ordering, one audit path | — |
| EscalationPolicyEntity | ENT | The per-agent low-confidence policy a person defines. Until it is set, that agent has no automatic escalation | Policy source of record |
| Views1queries that recompute when their sources change | |||
| RoutingTableView | VIEW | The read side the request path consults on every call | — |
One writer, by design. T4 asks on promotion, S3 asks when a model decays, a person asks on approval — and the gateway makes every change. That is what gives the routing table a single ordering and a single audit trail. A second writer would make rollback ambiguous exactly when it matters most.
Escalation is configuration. The low-confidence signal is whatever a person configured for that agent: an exact check where the task allows one, a confidence threshold, or a judge call. There is no default, and an agent without a policy simply never escalates automatically.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Request classification | Works out which agent and task shape a request belongs to | |
| Tier selection | Chooses the small model, the incumbent, or the frontier tier | |
| Escalation policy evaluation | Applies the per-agent rule that sends a hard request upward | |
| Routing-intent application | Applies asks in one order and emits what changed |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why this workflow needs it | Constraint |
|---|---|---|---|
| Small-model serving | Model inference | Answers the request when the routing table says the small model serves it | In-VPC or managed; the same OpenAI-compatible call either way |
| Adapter hot-swap | Model serving | Binds the right small adapter onto a resident base model | The adapter is small; the base stays loaded. This is what makes per-agent models affordable |
| Frontier provider | Model inference | Serves escalations and anything with no promoted model | Reached through the gateway, so it is recorded like everything else |
| Usage accounting | Telemetry | Returns tokens and latency on every response | This is the raw material for every cost number downstream |
Shadow & Canary
GatewayHow a new model earns real traffic. The gateway is the one place where a candidate and the model in service can answer exactly the same request, so the comparison is like for like.
Flowtwo phases · both answer the same request at once
solid = real traffic · grey dashed = a copy, or a result being read · red dashed = the way back · green = nobody is exposed
Trigger
T4 moves a candidate onto the shadow or canary step and sets the share of traffic it may take. This workflow carries that out; it does not decide it.
State & durability
The share is durable and versioned. Paired results carry a matching id, so a comparison made today can be reproduced from the record months later — the same request, the same moment, both answers.
Failure & compensation
Shadow answers are scored and thrown away, so no user ever receives one. If the candidate starts answering worse, the gateway puts everything back on the model in service immediately, then reports the breach to T4, which records the failed step and sets the candidate aside. Serving continues throughout.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| Shadow | POST /v1/chat/completions | the candidate answers a copy of the request; the answer is scored and discarded | an answer nobody receives | |
| Canary | POST /v1/chat/completions | the candidate answers its share of real requests, on the same endpoint as the model in service | an answer a real user receives | |
| Set or revert the share | — not an inference call | the gateway's own traffic policy, changed in place | confirmation of the new share | |
Components to buildevery component this workflow needs, and the scaffolding each one depends on
Where a candidate meets real traffic. The revert is inline and does not wait on the control plane — safety belongs to whatever is closest to the request.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Endpoints1the inline request path — no durable per-request state | |||
| GatewayEndpoint | EP | Shared with S1. Mirrors traffic for shadow, splits it for canary, and reverts in place on a breach | Split policy read on the request path |
| Consumers1react to a stream of changes | |||
| PairedScoringConsumer | CON | Scores paired captures and discards the shadow output, so no user ever receives one | Judge access · correlation ids |
| Entities2durable state, one instance per key | |||
| SplitPolicyEntity | ENT | Durable and versioned. What share of traffic a candidate may see — the exposure cap | — |
| PairedResultEntity | ENT | Both arms of a comparison under one correlation id, so a result is reproducible months later | — |
| Views1queries that recompute when their sources change | |||
| CanaryHealthView | VIEW | Acceptance metrics over the canary window — what the breach check reads | — |
S2 acts, T4 judges. On a canary dip S2 reverts the split to the incumbent immediately, without waiting for the control plane, and only then reports the breach. T4 owns the saga and records the verdict; S2 owns the safety action. Splitting it this way means the fast path is never blocked on a slower one.
Shadow costs money and shows nothing to users. Every shadow request is a second inference that is scored and thrown away. That is the price of a risk-free comparison, and it should be a deliberate, bounded spend rather than a permanent background cost.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Traffic mirroring | Duplicates live requests to the candidate without affecting the answer served | |
| Traffic splitting | Sends a bounded share of real traffic to the candidate | |
| Paired capture and correlation | Ties both arms of a comparison together for later reproduction | |
| Acceptance evaluation | Decides in-window whether the candidate is still within its bar | |
| Immediate revert | Returns all traffic to the incumbent without a control-plane round trip |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why this workflow needs it | Constraint |
|---|---|---|---|
| Model serving, both arms | Model inference | Runs incumbent and candidate over the same input | Shadow doubles inference cost for the duration |
| LLM judge | Model inference | Scores the paired results | Same judges as T4's bar, or the comparison is not like-for-like |
Drift Watch
Akka orchestrationA promoted model keeps being marked for as long as it serves, against the same scores it had to beat to go live in the first place. Where it slips sharply the system acts on its own; where it slips slowly, a person decides.
Flowon a timer · three ways it can end
yellow = waiting on a person · grey dashed = read · red dashed = the system acts on its own · green = nothing to do
Trigger
A timer, on a fixed cadence, for every promoted model — reading a rolling window of that model's live traffic. Nothing about it is event-driven, so a quiet agent is still checked.
State & durability
A rolling window per agent, durable across restarts. What the model proved at promotion is kept alongside it, so slipping is measured against the scores it actually had to beat.
Failure & compensation
A sharp drop is treated exactly like a failed canary: ask the gateway to go back to the previous version or to the frontier, and re-open training. A slow slide is a decision for a person — train it again, or return the agent to the frontier for good — and it keeps serving while that decision waits.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| Mark the answers | POST /v1/chat/completions | vLLM serving the judge model in your VPC | Fireworks, or the frontier TEACHER tier | a quality score for live answers, on the same scale T4 used |
| Watch the trend | — not an inference call | read from the record, not from the models — quality, cost per task, and how often S1 fell out to the frontier | the trend over the window | |
| Go back | — not an inference call | a request to the gateway, which is the only thing that changes what serves | confirmation of what changed | |
Components to buildevery component this workflow needs, and the scaffolding each one depends on
A promoted model is not finished. This watches it against the standard it actually proved at promotion, and separates a cliff from a slide — because they need different responses.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Workflows1durable multi-step processes — each resumes at its last completed step | |||
| DriftCheckWorkflow | WF | Grades a window of production responses, compares against the retained promotion bar, and renders an outcome | Judge access · retained evidence |
| Timed Actions1schedule only — each starts a run and does no work itself | |||
| DriftWatchTimedAction | TMR | Fixed cadence per promoted agent. Starts one drift check and does no grading itself | Self-rescheduling timer |
| Entities2durable state, one instance per key | |||
| RollingWindowEntity | ENT | The window of recent graded production traffic per agent, durable across restarts | — |
| PromotionEvidenceEntity | ENT | Shared with T4. The standard the model proved, retained so drift is measured against the real thing | — |
| Views1queries that recompute when their sources change | |||
| DriftByAgentView | VIEW | Current position against the standard it proved, per promoted agent | — |
A cliff and a slide are different events. Sharp degradation is handled without asking anyone: emit a revert intent and re-open training. Gradual drift is a person's decision — retrain, or return the agent to the frontier for good — and the agent keeps serving while that decision waits. Automating the second one would either revert too eagerly or too late.
Measured against the retained standard. The acceptance evidence from promotion is kept precisely so this comparison is against what the model proved. Re-deriving a baseline later would let the standard drift along with the model.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Rolling-window maintenance | Keeps a bounded, durable window of recent graded traffic per agent | |
| Drift classification | Separates sharp degradation from gradual slide | |
| Comparison against the retained bar | Measures against promotion evidence rather than a recomputed baseline |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why this workflow needs it | Constraint |
|---|---|---|---|
| LLM judge | Model inference | Grades production responses | The same judges T4 used at promotion, or the comparison is meaningless |
| Routing intent | Control plane | Asks the gateway to go back on sharp degradation | The gateway makes the change; this workflow only asks |
| Metrics record | Data substrate | Supplies cost and latency | Read from the record, not from the inference tier |
Govern & Capture
GatewayChecking and recording happen in one pass. Every request is checked before it leaves and written down after, at the same point — so neither can be skipped, and there is nowhere else to go.
Flowone pass · one way to be stopped
solid = invocation · grey dashed = written to the record · red dashed = stopped here
Trigger
Every call in both loops — serving traffic from S1 and training traffic from T2 and T3 — plus production logs arriving in batches through the forwarder. Rollouts route through here too, so there is no path around it.
State & durability
The record is added to and never edited. What each call was cleared for travels with it, so T1 can rule an agent out later without having to work out where the data came from a second time.
Failure & compensation
A call that is not allowed is stopped, or forced out to the audited frontier path. It is written down either way — a refused call is part of the record just as a served one is. If stripping fails, the record is dropped rather than written, so nothing unscrubbed reaches the store.
Inference-tier interfaces
| Called by | Endpoint | Your GPUs · you serve | Fireworks · they serve | What comes back |
|---|---|---|---|---|
| All four steps | — none of its own | this workflow calls no model. It wraps the calls the other workflows make, before and after, and everything it does is deterministic | nothing from a model | |
Checking, stripping, counting, and recording are all ordinary code. Listing them as inference-tier interfaces would suggest a model is involved in deciding whether a call is allowed, and none is.
Components to buildevery component this workflow needs, and the scaffolding each one depends on
The band around everything else. Every call in both loops transits this path — including rollout traffic — so there is no route that escapes metering, scrubbing or the record.
| Component | Type | Purpose | Scaffolding needed |
|---|---|---|---|
| Endpoints2the inline request path — no durable per-request state | |||
| GatewayEndpoint | EP | Shared with S1 and S2. Admits, scrubs, meters and emits, inline, before the model call | Policy source · sanitizer · spend ledger |
| LogForwarderEndpoint | EP | Batch ingest for agents that cannot be pointed at the gateway. Capture and audit only, with no steering | Trace ingest format |
| Consumers1react to a stream of changes | |||
| TraceCaptureConsumer | CON | Writes traces to the store with their governance tags attached | — |
| Entities3durable state, one instance per key | |||
| PolicyEntity | ENT | Consent, RBAC and tenancy for an agent — what admit checks against | Policy source of record |
| SpendLedgerEntity | ENT | Token spend against the agent's ceiling | — |
| TraceEntity | ENT | One append-only trace, carrying the governance tags that let T1 veto later without re-deriving provenance | — |
| Views2queries that recompute when their sources change | |||
| TracesByAgentView | VIEW | The substrate every T1 scorer reads | — |
| SpendByAgentView | VIEW | Spend and ceiling per agent — what D1 renders | — |
Scrub before persist, and drop on failure. PII and secrets are removed at the boundary, before anything is written. If scrubbing fails the trace is dropped rather than stored unscrubbed — the record is allowed to have a hole in it, but never a leak. This is the natural home for Akka's sanitizer, which masks matched patterns and can be called from any component.
Governance tags travel with the trace. Regulated status and training consent are written onto the record here, at capture. That is what lets T1 veto an agent months later without re-deriving where the data came from — and it is why a wrong tag at this step is a compliance failure rather than a scoring error.
Two onboarding paths, two levels of control. Pointing an agent at the gateway gives inline steering, enforcement and capture. The log forwarder gives capture and audit only — there is no steering until that agent is proxied, and the difference should be explicit to whoever is buying.
What we build vs. what we wire inwhich parts are our algorithms, and which are someone else's capability
Algorithms we buildno external service — this is the IP
| Capability | Node | Why it exists |
|---|---|---|
| Admission policy evaluation | Checks consent, RBAC and tenancy before a call proceeds | |
| Metering against a ceiling | Counts spend per agent and enforces the limit | |
| Trace emission and tagging | Writes the append-only record with governance tags attached |
Capabilities we wire insomeone else's, called for a named reason
| Capability | Kind | Why this workflow needs it | Constraint |
|---|---|---|---|
| Sanitizer | Platform capability | Masks PII and secrets at the boundary before anything is persisted | Akka's own, injectable into any component — not a third party |
| Identity provider | Governance | Supplies the identities RBAC and tenancy are checked against | Customer-owned |
| Trace ingest format | Data substrate | How forwarded logs arrive for agents that cannot be proxied | Open tracing formats; the customer's exporters decide the shape |
Standing policy requirements
Akka VerifyG1 runs per request as a workflow. These are standing requirements evaluated against the record on a schedule. Akka Verify reads the cost and quality record and enforces them:
- 1Model eligibility. A task type is served only by models at quality above a floor and cost below a ceiling.
- 2Lowest-cost routing. Route each task to the cheapest qualifying model, updated as candidates prove out.
- 3Budget guardrail. Throttle or fail over when projected spend exceeds a cap.
- 4Per-task ceiling. Cap tokens or cost per task; downgrade or halt on breach.
- 5Cost as a promotion criterion. Hold a candidate that passes quality but costs more than the incumbent.
- 6Re-evaluate on price drift. Reopen routing when frontier prices or a model's blended cost move past a threshold.
- 7Value realization. Report savings against the frontier baseline from the record.
Every policy, and who decides it
Akka VerifyThree kinds of control, and the difference is who acts. A guardrail fires on its own. An evaluation produces a verdict that something else acts on. A human decision holds the system until a named owner chooses — and every one of those is a queue somebody has to work.
D2 as a queue with an owner| Policy | Where | Kind | What happens when it fires |
|---|---|---|---|
| Training loop | |||
| Regulated task may not leave the audited model | T1 | GUARDRAIL | Agent is removed before scoring; reason recorded |
| Training-consent scope and data residency | T1 | GUARDRAIL | Agent is removed before scoring; logged data stays audit-only |
| Customer ring-fence — what may never be touched | T1 | GUARDRAIL | Agent is removed before scoring |
| Minimum gradeability and minimum distinct examples | T1 | EVALUATION | Renders KEEP_FRONTIER with the failing dimension named |
| Override a KEEP_FRONTIER verdict for a wanted agent | T1 | HUMAN | Agent stays on frontier until someone accepts the risk in writing |
| Contractual pin — "this customer's contract specifies a named model" | T1 | HUMAN | Agent is excluded from optimization until the pin is lifted |
| Order of the training queue under a grader budget | T1 | HUMAN | Ranked plan is proposed; nothing starts until the order is confirmed |
| Allowed base-model menu | T2 | GUARDRAIL | Bases outside the menu are never proposed |
| Choice of base on the size frontier — cheaper vs safer | T2 | HUMAN | A recommendation with a risk curve; the run waits for the pick |
| Training spend cap per run | T2 | GUARDRAIL | Run halts at the cap and reports where it stopped |
| Teacher distillation and synthetic data permitted | T2 | GUARDRAIL | Excluded data classes never enter the dataset build |
| Reversible actions run for real, irreversible are mocked | T3 | GUARDRAIL | Enforced by the sandbox topology, not by configuration |
| Which repositories and environments rollouts may use | T3 | GUARDRAIL | Sandbox provisioning refuses anything off the list |
| Quality against baseline and token reduction against target | T4 | EVALUATION | Candidate advances a rung, or is retired with its evidence |
| Canary traffic share | T4 | GUARDRAIL | Caps exposure at the configured percentage |
| Promotion of a high-blast-radius agent | T4 | HUMAN | Candidate holds at the last passed rung until approved |
| Promotion when cost improves and quality is only level | T4 | HUMAN | Held — the saving is real but nobody gains quality |
| Serving loop | |||
| Model eligibility — which models may serve which task | S1 | GUARDRAIL | Ineligible models are never routed to |
| Route to the cheapest qualifying model | S1 | GUARDRAIL | Routing table updates as candidates prove out |
| Per-task token and cost ceiling | S1 | GUARDRAIL | Downgrades the tier or halts the task on breach |
| Frontier fallback permitted for this agent | S1 | GUARDRAIL | Where it is off, the request fails rather than escalating |
| What counts as low confidence for escalation | S1 | HUMAN | Reviewed and set per agent — an exact check where the task allows one, a logprob threshold, or a judge call. Until it is defined the agent has no automatic escalation |
| Projected spend exceeds the period cap | S1 | HUMAN | Throttle and degrade, or accept the overspend — someone must choose |
| A candidate on live traffic starts answering worse | S2 | GUARDRAIL | Split reverts to the incumbent immediately; T4 records the verdict |
| Paired shadow scoring against the incumbent | S2 | EVALUATION | Produces the comparison T4 reads at the shadow rung |
| Marking live answers against the scores the model was approved on | S3 | EVALUATION | Produces the drift signal; acts on nothing itself |
| Sharp degradation of a promoted model | S3 | GUARDRAIL | Reverts to the prior adapter or to frontier without waiting |
| Gradual drift — retrain, or return to frontier for good | S3 | HUMAN | The agent keeps serving while someone decides whether it is still worth optimizing |
| Re-admission after repeated drift | S3 | HUMAN | A third failure stops automatic re-entry into T1 |
| Every request, both loops | |||
| Consent, RBAC, and tenancy on each call | G1 | GUARDRAIL | Blocks the route or forces the audited frontier path |
| PII and secrets removed before anything is stored | G1 | GUARDRAIL | On failure the trace is dropped rather than written |
| Hold a candidate that passes quality but costs more | G2 | EVALUATION | Blocks promotion on economics alone |
| Re-open routing when model prices move | G2 | EVALUATION | Recomputes the cheapest qualifying model per task |
| Savings reported against the frontier baseline | G2 | EVALUATION | Feeds the record behind every figure on the dashboards |
| Standing exception to any guardrail above | G2 | HUMAN | Granted with an owner and an expiry, or the guardrail stands |
Continuous Cost Governance
Read sideBudget owner
Am I within budget — and is the saving real?
counterfactual · 79% below
run rate goes $12K/mo → $47K/mo
Waiting on you
2 openAI owner
What is worth training next — and is what I promoted still holding?
grader exists for their task shape
| Agent | frontier → proved → now | $/task | State |
|---|---|---|---|
| claims-extract v5 | 0.93 → 0.96 → 0.96 | $0.0028 | holding |
| code-review v3 | 0.91 → 0.94 → 0.89 | $0.0041 | drifting |
| deploy-agent v2 | 0.88 → 0.92 → 0.92 | $0.0035 | holding |
worse than when it was approved — so S3 re-opens T1
Waiting on you
4 openSystem behavior
Is the loop running — and is it failing safe when it fails?
regression before it reaches the fleet
the serving tier's latency budget
Waiting on you
2 openDecisions waiting on a person
Read sideThe eleven human decisions in G3 are the only places the loop stops on its own. Each one holds something — a candidate, a saving, an agent's routing — and holding has a price. This view carries that price, so the queue is worked in the order the money says.
| Waiting on | Owner | Open | If nobody acts | Cost of waiting |
|---|---|---|---|---|
| Promote claims-extract v6 to the fleet passed all four steps · a mistake on this agent would be costly, so it needs sign-off | AI owner | 9d | Candidate holds at canary, serving 5% of traffic | $1,240 / mo |
| sre-triage has no rule for when an answer is too weak to send without one the agent never falls back to the frontier | Platform | 7d | Weak answers are served instead of escalating | fails open |
| Accept projected spend of $63.1K against a $60K cap driven by frontier fallback on code-review, which is drifting | Budget owner | 6d | Throttle engages at the cap; roughly 12% of requests degrade | 3.1K requests |
| code-review is answering worse than when it was approved — train it again, or return it still better than the model it replaced, but worse than it was on the day it went live | AI owner | 3d | Keeps serving at a widening quality gap | $890 / mo |
| Whether the claims-v2 data class may be trained on logged for governance today; not yet cleared as training data | Compliance | 2d | 19 agents stay ineligible and are never scored | $3,100 / mo |
| code-review went over its cost-per-task ceiling 14 times this week $0.0041 against a $0.0035 ceiling | Budget owner | 2d | It keeps serving, at the higher cost per task | $0.0006 / task |
| Which base model sre-triage is built on — cheaper, or safer 8B saves more and carries more quality risk; 32B is the safe point | AI owner | 1d | The training run does not start | start delayed |
| Order of the next training queue 6 admitted agents, grader budget covers 3 this cycle | AI owner | 4h | Nothing new enters training this cycle | 1 cycle |
decisions denominated in money
decisions denominated in quality and risk
decisions that settle whether an agent may run at all
A 90-day sequence
Connect
Deploy in the chosen environment, pass InfoSec, connect the datalink and trace sources.
Target
Targeting scores live traffic and returns a ranked plan with quality baselines.
First model
The first adapted model is trained, evaluated, and shadow-tested, with a before/after benchmark on your traffic.
The sequence is the same for both deployment options; the training and inference engine differs.
What you supply, what Akka runs
Inputs
- GPUs, or a Fireworks account.
- The private datalink to the environment.
- Persona and tool scopes for the loop.
- Evaluation definitions.
- Workspace access for training and testing.
The rest
- Capture, targeting, and the optimization workflow.
- Training orchestration and serving.
- Evaluations, gating, and rollback.
- The interaction record and Improvement Policy.