When coding agents overwhelm a self-hosted model, reject early

Harshul Jain of Audible and Parv Khatri of SAP introduce admission control: deciding which requests can enter a self-hosted language-model service when coding agents send bursts of work.

Inference begins with a prompt and produces an answer token by token. Jain emphasizes that the final output length is not known at the start, making the resources a request will consume difficult to predict.

A request passes through an application, an API gateway, a router and an inference engine such as vLLM or SGLang. The engine loads model weights onto GPUs, batches requests and maintains a key-value cache for each running request.

Jain describes a tension between quality, latency and throughput. Interactive chat may prioritize response quality and speed; asynchronous agents may tolerate more delay for throughput, while high-volume autocomplete can favor speed over a larger model.

Every deployment has a maximum load at which it can still meet its service-level objective. The speakers cite OpenAI’s 2023 routing incident and an Anthropic capacity incident as examples where load shedding or earlier capacity detection mattered.

They propose a decision at the front door: admit a request now, hold it for later, or reject it. The decision should happen before the request consumes GPU memory.

During prefill the engine reads the prompt and builds the cache; during decoding the cache grows as the model generates. Accepting a request commits resources for its lifetime, so admitting a long request affects the work already running.

When GPU memory fills, an engine may preempt, swap or fail work. These responses can increase latency and waste computation. Self-hosted deployments need to protect their own capacity rather than assume the inference engine can absorb any burst.

A coding agent can fan out into many file reads, tests and model calls, with retries layered on top. One user request can therefore create tens of model requests close together, unlike a person typing one message and pausing to read.

Khatri asks where admission belongs. Agents know their own calls, gateways know keys and quotas, routers see replicas, and engines see batches and cache. None necessarily combines request details with current capacity before accepting work.

A policy can act at ingress, at the engine’s queue, or during decoding. Later decisions can see more state but may require modifying and maintaining a fork of the scheduler; the speakers focus on ingress because it can work with an unmodified engine.

At ingress, the policy combines request information with system state and chooses admit, defer or reject. The request consumes no GPU memory until admitted, although the front-end infrastructure still has to handle it.

Useful request signals include input length, tenant, session and deadline. Engine signals include cache usage, queue age, cache hits and replica-pool load. They recommend using metrics that change within seconds, because a minute average can hide a brief fan-out.

Their example rules defer a prompt that does not fit available headroom or excess calls from a session, reject a tenant over its share, and reject work that cannot meet its deadline. Engine-side rules consider cache headroom, queue age, cached prefixes and pool saturation.

An HTTP 200 begins a stream but does not guarantee that the stream finishes. Khatri advises clients to monitor the stream, handle partial failures and close the connection when abandoning work so resources can be released.

For a temporary defer, their example returns 503 with a Retry-After value. The client should wait for that interval plus random jitter. Without coordinated delays, many retries can arrive together and turn an initial burst into an even larger one.

For rejection, they distinguish a 429 caused by a caller exceeding its share from a 529 used in their example for provider overload. Clients should reduce their own concurrency for the former; for the latter, wait, switch provider or park the work. Both need a finite retry budget and backoff.

Jain explains why counting requests is not the same as measuring cost. A fixed per-client rate limit supports fairness but does not track GPU pressure, while a fixed in-flight count treats a short request and a long one alike.

A policy has four design choices: the unit admitted, the signal observed, the rule converting the signal into a decision, and the resulting action. A deadline-based rule, for example, cannot work unless requests actually carry useful deadlines.

Their illustrative comparison shows two sessions with 20 calls each and room for only half. Admitting half of each may leave neither complete; admitting one full session lets it finish while the other receives a clear defer. The diagram is a scenario, not a measured benchmark.

Signals serve different goals. Cache pressure can protect memory, cache hits can identify reuse, queue depth can reveal waiting, deadline slack can protect latency objectives, and output-length estimates can help plan for long answers. Each can fail to reveal a different bottleneck.

The speakers contrast simple thresholds, two-signal gates, optimizers and formal deadline bounds. A more elaborate decision can cost more time; an optimizer taking seconds may react too slowly to dozens of calls arriving in two seconds.

They summarize four research policies and their reported gains, while warning that the papers use different models, engines and workloads. They do not present the figures as a leaderboard. In their assessment none cleanly covers online, whole-session admission on an unmodified engine.

Their decision tree starts with deadlines; otherwise it asks whether the KV cache fills at peak load. Cache and reuse signals may be useful when memory is the bottleneck; a queue-based threshold may be more useful when the queue grows but cache does not fill.

Khatri recommends benchmarking a policy against a measured cluster baseline. Replay the same realistic agent trace with and without the policy, including fan-out, retries and sessions, and sweep load from below to above measured capacity.

Their proposed score divides requests that meet the latency objective by all offered requests, including rejected ones. If a system rejects 95% and perfectly serves the rest, admitted-only attainment is 100%, while this goodput measure is 5%.

A useful report should disclose load relative to capacity, deadlines relative to unloaded latency, signal freshness, repetitions and variation, parameter provenance, whether rejected requests count, and the complete model, engine, hardware, workload and baseline.

The team built AdmitPerf as both an ingress library and a benchmarking harness. The same policy function is used for the benchmark and deployment, so a comparison can run against a real engine and GPU without changing the policy being evaluated.

Their workflow starts infrastructure, checks it is serving, benchmarks policies repeatedly, compares results and shuts the infrastructure down. They keep the deployment fixed across policy comparisons so provisioning changes do not confound the result.

A policy must declare the signals it requires. If cache usage is unavailable, the benchmark should stop rather than silently treating the missing value as zero and turning a cache threshold into an accept-everything policy.

On their small Qwen 2.5 0.5B and A10G test, the displayed table reports 80 requests at 15 per second and two repeats. With a maximum of two waiting requests, 63.1% were admitted and p95 time to first token fell from about 2.3 seconds to 0.95 seconds at the same reported goodput.

In that run the KV cache peaked around 0.5% while the queue grew to 24 requests. A cache-pressure rule would have missed the bottleneck. The team also provides a fake engine so the setup can be explored on a laptop without a GPU.