Migrate from OpenAI API to Self-Hosted LLMs: 2026 Cost Guide

GPT-5.4, the mid-tier flagship most teams actually call in production, costs $2.50 per million input tokens and $15.00 per million output tokens (OpenAI pricing). If you've already decided to migrate from the OpenAI API to a self-hosted LLM, or you're 80% of the way there and need the numbers to close the internal case, this post skips the buying debate and gets straight to the math: the exact token volume where self-hosting wins, the two-line code change that gets you there, and the specific things that break mid-migration if you don't test for them.

For the "should I even leave the API" decision, see GPT-6 vs self-hosted LLMs. This post assumes that decision is already made.

OpenAI API Pricing in 2026: What You're Actually Paying Per Token

OpenAI runs a three-tier pricing ladder as of mid-2026. GPT-5.5 is the top-tier flagship at $5.00 per million input tokens and $30.00 per million output tokens, with cached input at $0.50/M. GPT-5.4, the mid-tier model most production workloads actually run on, is $2.50/M input and $15.00/M output, cached at $0.25/M. GPT-5.4-mini, the budget tier, is $0.75/M input and $4.50/M output, cached at $0.075/M (OpenAI pricing).

Two things change your effective rate before you even get to self-hosting math. First, the Batch API cuts every one of these rates in half for asynchronous jobs: GPT-5.4 drops to $1.25/M input and $7.50/M output in batch mode. If your workload doesn't need a synchronous response, that's a free 50% cut with no infrastructure change. Second, if you're on a regional data-residency endpoint for a model released on or after March 5, 2026, OpenAI adds a flat 10% uplift on top of standard pricing (OpenAI pricing). Check which tier and which endpoint you're actually calling before you run any breakeven math; the difference between GPT-5.4 standard and GPT-5.4 with the residency uplift is real money at volume.

Most workloads send more input than output, so the number that matters is a blended rate, not either headline figure on its own. At an 80/20 input/output split:

Blended GPT-5.4 rate = (0.8 x $2.50) + (0.2 x $15.00) = $2.00 + $3.00 = $5.00/M tokens
Blended GPT-5.4-mini rate = (0.8 x $0.75) + (0.2 x $4.50) = $0.60 + $0.90 = $1.50/M tokens

At a more output-heavy 60/40 split, both rates climb:

Blended GPT-5.4 rate = (0.6 x $2.50) + (0.4 x $15.00) = $1.50 + $6.00 = $7.50/M tokens
Blended GPT-5.4-mini rate = (0.6 x $0.75) + (0.4 x $4.50) = $0.45 + $1.80 = $2.25/M tokens

Every crossover number below is built on these blended rates, uncached. If a meaningful share of your traffic hits a cached system prompt, your real rate is lower and the crossover point moves out further in OpenAI's favor. Run your own numbers off your actual input/output ratio before treating any of these as gospel.

The Token Volume Where Self-Hosting Beats the OpenAI API

The short version: on a single on-demand H100 SXM5 running a 70B model in FP8 with vLLM, self-hosting breaks even against GPT-5.4 at roughly 12M tokens/day (80/20 split). Against GPT-5.4-mini, a 70B model on one H100 never breaks even at an 80/20 split, no matter how much volume you throw at it, because the GPU's own cost per token at full utilization already runs above mini's blended rate. It only crosses over against mini at a more output-heavy 60/40 split, around 27M tokens/day. That distinction matters more than the headline crossover number, so read the mini section before you build a business case around it.

The Breakeven Formula

The mechanics are simple once you separate the two cost shapes. The OpenAI API bills linearly: cost scales with every token you send. A self-hosted GPU bills as a fixed cost per hour, flat regardless of how many tokens you push through it, up to the GPU's throughput ceiling. That gives you:

Self-hosted daily cost = GPU $/hr x 24   (flat, until you hit the throughput ceiling)
API daily cost = daily token volume (in millions) x blended $/M rate
Breakeven volume (M tokens/day) = self-hosted daily cost / blended API rate

The throughput ceiling matters because it caps how far the "flat cost" story holds. A single H100 SXM5 running a 70B model in FP8 with vLLM's continuous batching delivers roughly 400 tokens/sec at typical concurrency, about 1.44M tokens/hour, which works out to a ceiling of roughly 34.5M tokens/day before you need a second GPU (Spheron's OpenAI-compatible API guide). Past that ceiling, cost steps up in whole-GPU increments rather than staying flat, and you need to redo the math per GPU added. See AI Inference Cost Economics 2026 for the fuller FinOps derivation this formula is built on.

Live Spheron GPU pricing (fetched 7 Jul 2026): H100 SXM5 on-demand runs $2.5352875/hr per GPU, so a single H100 costs about $60.85/day flat, whatever your volume, up to that ~34.5M tokens/day ceiling. Divide the two and you get a per-token floor: $60.85 / 34.5M tokens = about $1.76/M tokens at full utilization. That floor number is the one to compare against each OpenAI tier, because it's the cheapest your self-hosted 70B can ever get on that hardware.

Pricing fluctuates based on GPU availability. The prices above are based on 7 Jul 2026 and may have changed. Check current GPU pricing → for live rates.

Breakeven Against GPT-5.4 (the mid-tier flagship most teams use)

At the 80/20 blended rate of $5.00/M tokens, the breakeven volume is:

Breakeven = $60.85/day / $5.00/M = 12.17M tokens/day

At the more output-heavy 60/40 split ($7.50/M blended), it drops to 8.11M tokens/day. Both numbers sit comfortably inside a single H100's ~34.5M tokens/day ceiling, so this is a clean, single-GPU crossover: no second card, no cluster math, just one instance running flat-out below its ceiling.

Above 12.17M tokens/day, every additional million tokens you process is free on the self-hosted side and $5 more on the API side, right up until you outgrow the GPU and need to add a second one.

Breakeven Against GPT-5.4-mini (budget-tier comparison)

This is the one teams get wrong, so read it before you write the business case. GPT-5.4-mini's blended rate at 80/20 is $1.50/M tokens, which is lower than the $1.76/M floor a 70B model can hit on a single H100 at full utilization. That means at an 80/20 split, self-hosting a 70B model on one H100 never beats GPT-5.4-mini on raw compute cost, at any volume. Adding a second or third H100 doesn't change the ratio; it just replicates the same $1.76/M floor at larger scale.

The math flips at a 60/40 split, where mini's blended rate rises to $2.25/M, above the $1.76/M self-hosted floor:

Breakeven (60/40 vs mini) = $60.85/day / $2.25/M = 27.04M tokens/day

That 27M tokens/day is still inside the single-H100 ceiling, so it's a valid, single-GPU crossover, just one that only shows up on output-heavy workloads.

The underlying reason is worth internalizing: mini tiers are cheap because they run a small model, not because OpenAI is discounting the same compute. Comparing a self-hosted 70B against a budget-tier API is not apples to apples unless your quality bar genuinely requires 70B-class output. If it doesn't, the better comparison is a smaller self-hosted model on cheaper hardware. A 7B model on an A100 80G SXM4 (on-demand $1.7952/hr live as of 7 Jul 2026) running at roughly 3,300 tokens/sec, a throughput figure consistent with vLLM's continuous batching on that hardware, works out to about $0.15/M tokens, which clears GPT-5.4-mini's $1.50/M rate at any meaningful volume. Pick the self-hosted model size to match the API tier you're actually replacing, not the model size you'd prefer to run.

The Labor Cost Most Teams Forget to Model

The GPU rate is not the whole bill. A self-hosted LLM deployment needs ongoing engineering time for monitoring, dependency upgrades, and incident response, and one documented single-GPU cost breakdown puts that at around $200/month (mlpivot.dev). That works out to roughly $6.67/day, which nudges your effective daily cost from $60.85 to about $67.52. Run the GPT-5.4 breakeven again with that number and the crossover moves from 12.17M to about 13.5M tokens/day, still comfortably inside the single-GPU ceiling. Treat $200/month as a floor, not a ceiling: it's the cost for a straightforward single-GPU setup, and a more complex, multi-tenant, or higher-availability deployment will need more engineering attention than that.

Reaching a working deployment in the first place doesn't take long: most teams get a single-node vLLM setup running in days (Introl). Getting from there to something production-hardened, meaning load testing, autoscaling, and observability, typically takes another one to two weeks of focused engineering work on top of that (Introl). And the labor bill doesn't stop at initial setup: growing past a single GPU means redoing capacity planning, load testing, and observability for the new topology, not just adding hardware. Scaling cost is part of the migration bill, not a one-time setup fee you pay once and forget.

Step by Step: Swapping base_url for a Self-Hosted vLLM Endpoint

The actual code change is small. The part that takes real time, and the part Spheron's full vLLM build guide covers flag-by-flag, is getting a production-grade endpoint behind it. This walkthrough stays narrow: just the moves that matter when you're cutting over existing OpenAI traffic rather than building a vLLM server from scratch.

Step 1: Provision the GPU and Pick a Model

Match the GPU to the model size you're replacing, not the model size you'd like to run. If you don't already know which open-weight model clears your quality bar, the VRAM tier guide walks through what fits at each VRAM tier.

Live Spheron on-demand pricing, fetched 7 Jul 2026:

Rent an H100 on Spheron for a 70B-class replacement, or an A100 for the smaller models that actually compete with GPT-5.4-mini. Provisioning with SSH root access takes under two minutes; the Spheron LLM quick-start docs cover the console steps if this is your first deployment. Confirm the GPU and VRAM you expect show up in nvidia-smi before installing anything.

Step 2: Start vLLM's OpenAI-Compatible Server

Install vLLM and start it with --dtype fp8 on H100/H200 (A100 has no native FP8 tensor cores, so use --dtype fp16 there instead), a --max-model-len sized to your KV cache budget, and --gpu-memory-utilization left with headroom. That's the same starting command the full build guide walks through flag by flag, so it isn't repeated here.

The one flag migration teams skip and then regret is --served-model-name. It sets the model string vLLM answers to in the API, and defaults to your --model argument if you don't set it (vLLM CLI reference). If your app currently sends "model": "gpt-5.4", pass --served-model-name gpt-5.4 when you start vLLM instead of hunting down every call site that hardcodes the model string. It's the difference between a two-line diff and a grep-and-replace across your codebase.

For production hardening (systemd unit, API key handling, NGINX with TLS), see the OpenAI-compatible API build guide, and for multi-GPU tensor parallelism once you outgrow one card, vLLM production deployment 2026.

Step 3: Change Two Lines in Your Existing Code

This is the entire application-level migration. vLLM's server implements the same protocol as OpenAI's, so your SDK calls don't change shape, only two values do: base_url and api_key (vLLM docs). The model field has to match whatever model name vLLM loaded.

from openai import OpenAI

# Before
client = OpenAI(api_key="sk-...")

# After: only base_url and api_key change
client = OpenAI(
    base_url="http://YOUR_GPU_IP:8000/v1",
    api_key="token-abc123",  # any non-empty string; vLLM doesn't validate keys by default
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",  # must match the model vLLM loaded
    messages=[{"role": "user", "content": "Summarize this contract."}],
    max_tokens=500,
)

Streaming works unchanged too; vLLM handles it via the same server-sent events format OpenAI uses for standard chat completions. Tool calling is the exception, covered below, so don't assume it's covered by "unchanged" until you've tested it specifically.

Step 4: Load Test Before You Cut Over Traffic

Don't point production traffic at a vLLM endpoint you haven't benchmarked. Run the same prompt set against both the OpenAI API and your new vLLM endpoint and compare, not just single-request latency, but throughput under realistic concurrency:

python vllm/benchmarks/benchmark_serving.py \
  --backend openai-chat \
  --base-url http://YOUR_GPU_IP:8000 \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --dataset-name sharegpt \
  --num-prompts 200 \
  --request-rate 10

Track TTFT (time to first token) at p50/p95, inter-token latency, and aggregate tokens/sec. A single request looking fast in isolation tells you nothing about what happens at 50 or 100 concurrent users; continuous batching means throughput climbs with concurrency up to a point, then queue depth starts showing up in your p95 numbers. Tune --max-num-seqs and --gpu-memory-utilization based on what the load test actually shows, not what worked in a single-user test.

What Breaks During Migration and How to De-Risk It

The two-line code change works for the majority of a typical workload. These are the parts that don't move over cleanly, and testing for them before cutover is what separates a smooth migration from an incident.

Tool Calling and Streaming Are Not Bit-for-Bit Identical

vLLM's tool calling is not on by default. You need --enable-auto-tool-choice and a matching --tool-call-parser for your model's format, and the tool_choice: "required" option specifically needs vllm>=0.8.3 (vLLM docs). If your OpenAI integration currently relies on tool_choice: "required" to force a function call, verify your vLLM version supports it before you cut over.

There's also a real, documented streaming bug worth testing for directly: when tool_choice names a specific function, vLLM's streamed tool call chunks can omit the "type": "function" field that OpenAI's schema expects, and some versions return arguments as None on the streamed response even though the non-streamed equivalent returns them correctly (vLLM issue #16340). Oddly, tool_choice: "auto" doesn't have the same issue, only the named-function path does. If any client library in your stack does strict schema validation on tool call responses (several do), a silent break here shows up as failed tool calls in production, not a clean error at migration time. Test streamed tool calls with your actual client library, not just a raw curl request, before you trust this path with real traffic. vLLM's own docs recommend defining your tool schemas in OpenAI's strict-schema style (additionalProperties: false, every property marked required, optional fields modeled as nullable) for the most reliable behavior across both streaming and non-streaming paths.

Output Quality Gaps Between GPT-5.4 and Open-Weight Models

Cost math doesn't account for whether the open-weight model actually holds up on your workload. Quantization is the biggest lever here: FP8 on H100/H200 hardware is close to lossless for most tasks, but pushing down to INT4/AWQ to fit a bigger model or cut GPU count introduces real, workload-dependent quality loss. If you're leaning on aggressive quantization to hit a cost target, the AWQ quantization guide covers what that tradeoff actually costs on benchmarks, not just VRAM.

Run your own eval set against both the current OpenAI model and the candidate open-weight model before cutover, on your actual task, not a generic benchmark. A model that scores well on public leaderboards can still underperform on your specific prompt patterns, especially for structured extraction, long-context reasoning, or domain-specific classification where GPT-5.4 has an edge that doesn't show up in a synthetic eval.

No More Automatic Uptime, Scaling, or Model Updates

The OpenAI API abstracts away three things you now own directly: uptime, capacity, and version upgrades. On a single self-hosted GPU with no failover, a hardware failure, driver crash, or configuration error takes your endpoint down until someone fixes it, with nothing automatically routing around the outage. That's the argument for at least a two-instance setup behind a load balancer before you route any latency-sensitive production traffic through a self-hosted endpoint exclusively.

Scaling isn't automatic either. If your token volume grows past a single GPU's ceiling, you provision, configure, and load-balance the next instance yourself, redoing the capacity planning and load testing described above for the new topology. And model quality doesn't improve under you the way it does with a managed API; if OpenAI ships a better GPT-5.x, that improvement lands in your app automatically. Once you self-host, upgrading to a newer open-weight model is a deployment you have to plan, test, and execute yourself.

A Phased Cutover Plan (Shadow Traffic, Then Percentage Rollout)

Don't flip 100% of production traffic to the new endpoint on day one. A workable sequence:

1. Shadow traffic. Send a copy of real production requests to the vLLM endpoint without using its response. Compare output quality, latency, and error rate against the live OpenAI responses for at least a few days of real traffic patterns, not a synthetic test set.
2. Low-percentage rollout. Route 5-10% of live traffic to the self-hosted endpoint, with an immediate rollback path if error rates or quality complaints spike. Watch tool-calling paths specifically here, since that's where the silent breaks described above tend to surface first.
3. Ramp by cohort, not by percentage alone. If you can segment by use case (say, internal tooling before customer-facing chat), migrate the lower-risk cohort first and use it to validate load-test assumptions against real concurrency.
4. Full cutover with the OpenAI path kept warm. Once the self-hosted endpoint has run at 100% of a cohort's traffic without incident for a sustained period, complete the rollout. Keep the OpenAI integration code in place (even if unused) for at least one release cycle as a fallback if a GPU incident takes your instance down before your redundancy plan is fully built out.

Should You Migrate at All? A Quick Decision Checklist

Run the numbers before you commit engineering time to a migration that might not pay off:

If you're also moving off a hyperscaler cloud at the same time as the API, the migration playbooks share a shape (audit, containerize, load test, phased cutover): see migrating from AWS, GCP, and Azure to alternative GPU clouds. And if you're weighing this same decision against Anthropic's API instead of OpenAI's, the token math works the same way; see Claude Opus 4.8 API vs self-hosted LLMs for that side-by-side. For teams already running vLLM but migrating off a different serving stack, migrating from Hugging Face TGI to vLLM or SGLang covers the equivalent flag-by-flag translation for that move.

If you're clearing double-digit millions of tokens a day against GPT-5.4 and the labor math still pencils out, the next step is provisioning the GPU and running your own load test against real traffic.

Check H100 availability → | View current GPU pricing →