SGLang Decisions API Explained: Serving Models as Classifiers (2026)

SGLang shipped a new endpoint in version 0.5.21 that answers a classification question in a single forward pass with no text generation at all. It's called the Decisions API, it lives at /v1/decisions, and SGLang's own release notes describe it as a way to turn an LLM or VLM into "a low-latency classifier and scorer." If your pipeline uses an LLM to route a request, moderate a message, or grade an answer, this changes how much GPU that step actually needs, and vLLM currently has nothing that does the same job.

TL;DR: What Is SGLang's New Decisions API?

SGLang's Decisions API, shipped in v0.5.21, is an endpoint at /v1/decisions that scores a model's answer-token probabilities in a single forward pass instead of generating text, turning any chat-template model into a classifier.

• What changed: v0.5.21 added /v1/decisions for typed choice (2-26 options), score (2-10 levels), and yes/no questions, scored from logits instead of decode, per SGLang's release notes.
• No special model needed: any chat-template model works, per SGLang's decision models docs.
• Latency: at 90 questions/sec on Qwen3-8B, SGLang's shared-query path holds roughly 130ms p95 versus seconds on Generate, per LMSYS's benchmark.
• vLLM gap: no equivalent endpoint as of 29 Oct 2026.
• GPU consequence: skipping decode lets Spheron's SGLang deployments run classification on a smaller, per-minute-billed GPU. See current GPU pricing.

What the SGLang Decisions API Does: Classification Without a Full Generation Pass

A normal LLM call to classify something, say, "is this message spam: yes or no", still runs the full generate loop: prefill the prompt, then decode token by token until the model writes out "yes" or "no" (or a longer explanation if you didn't constrain it). That decode loop is where most of the latency and most of the GPU cycles go, and it's wasted work when the only thing you actually need is which of two tokens was more likely.

The Decisions API skips it. According to SGLang's documentation on decision models, the server renders your question through the model's own chat template with thinking disabled, then scores the answer-label tokens through the same prefill path used by SGLang's existing /v1/score endpoint, and normalizes the result with softmax. The response includes a label_mass field that reports how much of the full-vocabulary probability landed on the labels you offered, which is a built-in confidence signal: a high label_mass means the model was genuinely choosing between your options, and a low one means most of the probability mass went somewhere else, a sign the question or label set might be miscalibrated.

Three request shapes are supported, per the same documentation:

• Typed choice: 2 to 26 labeled options (think routing a ticket to one of several queues).
• Score: 2 to 10 ordered levels (think a 1-5 quality or toxicity rating).
• Yes/no: a binary decision (think a pass/fail moderation gate).

No special checkpoint is required. Any generation model that SGLang serves with a Jinja chat template can answer Decisions API requests, so an existing deployment doesn't need a dedicated classifier model bolted on. There are real constraints, though: the route refuses models using built-in conversation templates, --enable-mis, or --dllm-algorithm, and the docs note that labels must be single tokens. For yes/no questions specifically, label_mass only counts lowercase tokens, so it reads lower by construction than the typed-choice or score variants, which is worth knowing before you set an alerting threshold on it. If you need the same answer twice from the same cache state, SGLang's docs point to --enable-deterministic-inference for reproducible scoring.

SGLang vs vLLM: Why Only One of Them Ships a Decisions API

Here's the direct answer to the question that actually gets searched: as of SGLang's v0.5.21 release, vLLM has no dedicated endpoint for typed classification requests. vLLM exposes logprobs through its completions API, which is enough to build equivalent logic yourself, but there's no single route that renders a chat template, restricts scoring to a label set, and reports a confidence figure the way /v1/decisions does out of the box.

This is a narrower claim than the usual "SGLang vs vLLM" comparison, which tends to argue architecture: RadixAttention against PagedAttention, prefix cache design, scheduler internals. Our own vLLM vs SGLang RadixAttention and PagedAttention benchmark post covers that ground, and if you're weighing vLLM, TensorRT-LLM, and SGLang generally for a new deployment, the LLM inference optimization decision framework walks through that choice end to end. This post is about one specific, dated feature gap: SGLang ships an endpoint purpose-built for "classify this input" with no generation step, and vLLM doesn't, as of this release.

Why This Matters for Routing, Moderation, and Guardrail Workloads

Routing, moderation, and guardrail checks are exactly the kind of request a generation-oriented serving stack handles inefficiently. Each one is a yes/no or small-multiple-choice decision riding on top of a serving path built for open-ended text. Our LLM inference router guide already covers the GPU math for sizing a routing tier that sends simple queries to cheap models and complex ones to larger models; the Decisions API changes that math further, since the router's own classification step no longer needs a decode loop to make its call.

The latency numbers back this up directly. At 90 questions/sec offered load on Qwen3-8B, SGLang's shared-query scoring path (the mechanism the Decisions API builds on) holds roughly 130ms p95 latency, while the same load through the single-item Generate path climbs into seconds, according to LMSYS's benchmark post on scaling decision models with SGLang. For a guardrail check that has to clear before a response reaches a user, that's the difference between a classifier that fits inside a request's latency budget and one that becomes the bottleneck.

The Rust-Core Prefix Cache and Restart-Free Prefill/Decode Role Switching

The Decisions API didn't ship in isolation. The same v0.5.21 release made SGLang's prefix cache run on a Rust core by default, with an automatic fallback to the Python implementation for unsupported configurations, or a forced fallback via the SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python environment variable, according to the release notes. That release also let prefill-decode (PD) disaggregated instances switch roles on the fly, without a restart. The release notes report concrete throughput gains from these changes: DeepSeek-V4.1 got 22% faster first-token latency on long prompts, and Kimi K3 got 20.6% higher prefill throughput in PD serving.

The restart-free role switching matters for classifier workloads specifically: a fleet running the Decisions API alongside a normal generation service can shift GPUs between prefill-heavy classification traffic and decode-heavy generation traffic as the mix changes through the day, without taking an instance offline to reconfigure it.

The same release tightened a security default worth knowing about if you're upgrading: a request-supplied chat_template inside chat_template_kwargs is now rejected unless the server is started with --trust-request-chat-template. It also removed the /get_weight_version, /weight_version, and /get_load routes in favor of /model_info and /v1/loads, so any monitoring tooling pointed at the old routes needs updating before you upgrade.

This isn't the first time an SGLang point release has flipped a default that affects GPU spend. Our post on SGLang's breakable CUDA graphs becoming the default prefill path in v0.5.15 covers the prior one, and it's a reasonable habit to check release notes before every production upgrade rather than after something breaks.

GPU Sizing: What a Classifier-Only Workload Needs vs Full Generation

A classifier-only workload through the Decisions API doesn't run the decode loop, so the usual GPU sizing logic for generation workloads, built around sustaining a decode throughput target across concurrent users, doesn't directly apply. What matters instead is prefill compute and KV cache capacity for short prompts, since every request is a single scored forward pass rather than a multi-second stream of output tokens.

That's a real downsizing opportunity for a specific slice of the stack: the classification step inside a router, moderation layer, or guardrail check, not the generation model behind it. A team already running a generation model on an H100 or H200 doesn't need to put the classifier on the same tier. Our NVIDIA L40S for AI inference post covers the specs and benchmarks for a GPU that's commonly right-sized for 7B-13B inference, and a classification-only pass against a model in that range is a workload the L40S's 48GB of GDDR6 and FP8 tensor cores handle comfortably, without a multi-second decode loop competing for the same memory bandwidth.

That per-minute billing matters here: a classifier tier sized separately from the generation tier behind it doesn't need a separate contract or provider relationship, just a smaller instance on the same marketplace. One honest limit worth naming: Spheron's SGLang quick-guide is an overview page and doesn't yet document the /v1/decisions request and response format itself, so you'll still want SGLang's own documentation open for the API contract, since v0.5.21 is new enough that provider docs generally haven't caught up to it.

How to Call the Decisions API: Request Format and a Worked Example

The Decisions API accepts a question, a list of labels, and the question type. A typed-choice request routing a support ticket to one of three queues looks like this against an SGLang server running a Jinja-chat-template model:

curl -X POST http://localhost:30000/v1/decisions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-name",
    "question": "Which team should handle this ticket: My payment was charged twice?",
    "question_type": "choice",
    "labels": ["billing", "technical", "account"]
  }'

The response carries the chosen label, the softmax-normalized probability for each offered option, and the label_mass figure showing how much total probability landed on the three labels versus everything else the model could have said. A low label_mass is a useful signal on its own: it means the ticket didn't cleanly fit any of the three queues, which is worth routing to a human rather than trusting the top label blindly.

For a yes/no guardrail check, the shape is simpler: a question_type of "yes_no" and no labels array needed. Remember that label_mass on yes/no questions only counts lowercase tokens per SGLang's documentation, so it reads lower than a typed-choice request by construction, not because the model is less confident. Set monitoring thresholds per question type, not as one blanket number across the whole endpoint.

Classification is a smaller, cheaper problem than generation, and until v0.5.21 most serving stacks made you pay generation prices for it anyway. SGLang's Decisions API is a narrow feature, but it's the kind of narrow feature that changes a line item on a GPU bill: a routing or moderation layer no longer has to share a GPU tier with the generation model it's gatekeeping. If you're deploying SGLang on Spheron already, adding the Decisions API to an existing endpoint doesn't require switching providers, and a classifier-only tier is a candidate to try on a cheaper GPU than whatever is serving your main model.

Running SGLang for routing, moderation, or guardrail checks is a different sizing problem than running it for generation, and the GPU catalog should reflect that.

Deploy SGLang on Spheron → | Rent an L40S → | View all GPU pricing →