Local LLM vs Cloud API: The Break-Even Math

You buy a GPU to stop paying per token. The card arrives, the model loads, the chat works, and it feels like free inference forever. Then a month passes and you cannot say whether you saved anything, because nobody hands you a bill for electricity and idle hours.

This article does that math with real numbers. Every price was pulled live on October 6, 2026: token prices from OpenRouter's public model API, GPU rental rates from the vast.ai and RunPod pricing pages, checkpoint sizes from the Ollama library. The 2026 answer is more interesting than the 2024 one, because the open-model market split in two, and the break-even now points in opposite directions depending on which half your workload lives in.

Tl;dr

• Cloud APIs charge per token with zero fixed cost. A local GPU bills by the hour whether it answers queries or idles. The comparison is a utilization problem, not a price problem.
• New premium open models flipped the math in your favor. Qwen3.8-27B costs $0.425/$2.55 per M on the API, but its 18GB Q4 checkpoint fits in a single 24GB card. A rented RTX 4090 breaks even at ~120M tokens/month, roughly 47 tok/s sustained. That is reachable.
• Commodity open models are near-unbeatable. gpt-oss-120b at $0.037/$0.17 would need ~3,100 tok/s sustained on a rented 96GB card to break even. Gemma 4 31B needs ~340 tok/s.
• Big open models lost their local case. Llama 4 Maverick (400B MoE, 245GB Q4) needs 3-4 datacenter GPUs, ~$5,700/month rented, break-even near 10.7B tokens/month.
• Closed frontier (Claude Sonnet 5.5, GPT-6 Astra, Gemini 3.8 Flash) has no local option at any budget. "Local vs cloud" is really "open weights vs everything."

Why the two pricing models don't compare

An API price is marginal cost: you pay when a token is generated, and $0 between requests. A local GPU is fixed cost: it bills by the hour while a checkpoint sits in VRAM, queries or no queries.

That difference means every "local is 10x cheaper" claim quietly assumes 100% utilization, and every "API is cheaper" claim assumes the hardware idles. We state assumptions explicitly in each example below so you can swap in your own.

One framing tool: blended price. Input and output tokens are priced very differently, so we use a 1:3 input:output bundle, a common shape for agent and RAG traffic. Output-heavy workloads (summarization, generation) push the API side up; input-heavy workloads (classification, extraction) push it down. We covered this trap in a previous post on cost per million tokens.

What local inference actually costs

Own it, or rent it.

Owning: cash costs are electricity plus depreciation. A 4090 system draws ~550W under load (450W GPU per NVIDIA's spec, ~100W system). Running 24/7 is 396 kWh/month; at an example $0.12/kWh, substitute your rate, that is $47.52/month. Depreciation is your call, but the card is not free.

Renting removes capex, and the rates are public. Pulled October 6, 2026:

Sources: vast.ai/pricing, runpod.io/pricing. Both are marketplaces; rates move with supply. Treat this as a snapshot, not a quote.

What the API charges for the same models

Every model people actually self-host is open-weight, and every one is served on API marketplaces. OpenRouter live prices, same day, $ per million tokens, with Q4 checkpoint sizes from Ollama:

Notice the spread: five to fifteen times between the cheapest and most expensive per token, for models that run on similar iron. That spread is the whole story.

The break-even, worked through

Formula: hardware $/month ÷ bundle price = bundles/month; × 4M tokens = tokens/month; ÷ 2.59M seconds = sustained tok/s required just to tie.

def break_even(hw_usd_per_month: float, price_in: float, price_out: float) -> tuple:
    """Return (tokens_per_month, sustained_tok_s) needed to tie the API."""
    bundle = price_in + 3 * price_out   # $ per (1M in + 3M out)
    bundles = hw_usd_per_month / bundle
    tokens = bundles * 4_000_000
    return tokens, tokens / 2_592_000   # seconds in a 30-day month

# Qwen3.8-27B on OpenRouter: $0.425 in / $2.55 out
# RTX 4090 on RunPod community: $0.34/hr -> $244.80/month
print(break_even(244.80, 0.425, 2.55))
# (121_113_300.9..., 46.7...)  -> ~121M tokens/mo, ~47 tok/s sustained

Qwen3.8-27B — the local case that now works. 18GB checkpoint, fits one 24GB card with context room. API bundle $8.08:

Forty-seven to sixty-five tokens/second, around the clock. A batched vLLM deployment on a 4090 serving a 27B Q4 model plausibly reaches that under concurrent load; single-stream llama.cpp will not. We did not measure it ourselves, so benchmark your exact stack; Artificial Analysis publishes independent throughput data. The point is the order of magnitude: for a model that's six months old and priced as premium, a busy small team can now genuinely beat the API.

Gemma 4 31B — borderline. Bundle $1.11. Same rented 4090 ($245/mo): break-even 880M tokens/month, ~340 tok/s sustained. Only real production traffic gets there.

gpt-oss-120b — don't bother. 65GB needs a 96GB card (RTX PRO 6000, vast median $1.55/hr = $1,116/mo). Bundle $0.55: break-even 8.2B tokens/month, ~3,100 tok/s. The cheapest models to rent are the hardest to beat, because marketplaces serve them on amortized fleets near marginal cost.

Llama 4 Maverick — the local case is dead. 245GB Q4 means three 96GB cards or four H100s. RunPod 4×H100 PCIe at $1.99 = $5,731/mo. Bundle $2.14: break-even 10.7B tokens/month, ~4,100 tok/s. At that volume you negotiate enterprise rates anyway.

Closed frontier — no comparison exists. Claude Sonnet 5.5 ($2/$10), GPT-6 Astra ($10/$50), Gemini 3.8 Flash ($0.75/$3.75): weights not released, no local option at any budget. If your product needs them, the local column is empty and the real question becomes which gateway and which open-weight fallbacks.

The practical shape of that decision is two base URLs and one model string. Same OpenAI SDK, local today, cloud tomorrow:

# Local: Ollama serving qwen3.8 (18GB Q4, fits one 24GB card)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8", "messages": [{"role": "user", "content": "hi"}]}'

# Cloud: same wire format, swap base URL and model string
curl https://api.anyapi.ai/v1/chat/completions \
  -H "Authorization: Bearer $ANYAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen/qwen3.8-27b", "messages": [{"role": "user", "content": "hi"}]}'

Because the request format is identical, benchmarking the same workload on rented iron for a week is a config change, not a rewrite.

And utilization, the silent killer: every number above assumes the GPU works 24/7. Four hours of interactive use a day multiplies your effective cost per token by 6. Idle weekends, by 10.

What the break-even hides

Concurrency. API providers batch hundreds of users per GPU; that is how a 120B model costs $0.14 per blended M and the provider still earns. Your single card serves your single queue unless you build batching, which is real engineering.

Quantization. Sizes above are Q4 defaults, the format local stacks ship. Quantization trades accuracy for memory, and how much depends on your workload. Run the quantized checkpoint against your own evals before declaring victory.

The MoE discount. Llama 4 Scout and Qwen3.5's large variants are mixture-of-experts: 109B and 397B total parameters but 17B active per token. They run far faster than dense models of their size, which quietly improves the local side of the table for exactly the models that need the most VRAM. VRAM is still the wall: Scout's 67GB does not fit in 48GB no matter how few experts fire.

Operations. Drivers, CUDA versions, engine upgrades, new releases every few weeks, 60-250GB downloads. An API is a URL. Self-hosting is a service you now run.

When local wins regardless of math

• Data governance. Prompts never leave the building. For healthcare, legal, finance, or any workload where a third-party subprocessor is a compliance problem, local is the only option, and no API discount changes that.
• Offline and edge. Air-gapped environments, ships, factories, places where the network is the failure mode.
• Custom fine-tunes. Your LoRA on an open base, served exactly as tuned, with no dependency on a provider keeping your endpoint alive.
• Single-user latency. No network hop, no shared queue. An idle local card can beat a loaded API endpoint on time-to-first-token even when it loses on throughput.
• Saturated hardware you already own. If the GPU exists and runs hot anyway, marginal inference cost is electricity only: the ~9 tok/s row above.

The middle ground

Between "buy a 4090" and "pay per token" sits hourly rental, cheaper than most assume. A 3090 at vast.ai's from-price of $0.09/hr is $65/month and runs 24GB models today. Rent for a week, benchmark your real workload, and replace every guess in this article with your own numbers. The experiment costs less than a dinner.

On the API side, the middle ground is a gateway: one endpoint spanning open-weight and closed models, with automatic fallback, so a provider outage or rate limit does not become yours. That is what we built AnyAPI to do: one OpenAI-compatible base URL, routing and failover at the gateway, and Qwen3.8, Gemma 4, Llama 4, and gpt-oss in the same catalog as the frontier models you cannot self-host at all.

Conclusion

Decide with two numbers: monthly token volume and duty cycle. For premium-tier open models like Qwen3.8-27B, break-even now sits at ~120M tokens/month on rented iron, reachable for a busy small team, and local wins above it. For commodity open models like gpt-oss and Gemma 4, the API prices are near marginal cost and you will not beat them until you own a fleet. For closed frontier models there is nothing to compute. Governance overrides all three rows: if prompts cannot leave, you do not have a pricing problem. Either way, test on rented iron first. At 2026 marketplace prices, guessing is the only losing move.

Frequently asked questions

Is it cheaper to run LLMs locally in 2026?

For premium open models, increasingly yes: Qwen3.8-27B breaks even at ~120M tokens/month on a rented RTX 4090. For commodity open models (gpt-oss, Gemma 4), no: API prices are near marginal cost and break-even needs 300-3,100 tok/s sustained. For closed models, the question does not apply.

What GPU do I need for current open models?

24GB (RTX 3090/4090) covers qwen3.8-27b (18GB Q4) and gemma-4-31b (19GB). 96GB (RTX PRO 6000) covers gpt-oss-120b (65GB) and llama-4-scout (67GB). Llama 4 Maverick (245GB) needs 3-4 datacenter GPUs.

Can I run Claude Sonnet 5.5 or GPT-6 locally?

No. Those weights are not released. The local column covers open-weight models only: Qwen3.8, Gemma 4, Llama 4, gpt-oss, GLM, DeepSeek, Kimi, MiniMax.

Are MoE models cheaper to self-host?

Faster, not smaller. Llama 4 Scout activates 17B of 109B parameters per token, so throughput behaves like a small model, but the full 67GB checkpoint must sit in VRAM. Compute gets cheaper, memory does not.

Is local inference faster?

For one interactive user on idle hardware, time-to-first-token can be lower, no network hop, no shared queue. Sustained throughput is different: API endpoints run batched serving stacks on datacenter fleets, which is why gpt-oss-120b costs $0.55 per blended bundle.

References

1. OpenRouter model list API (token prices, pulled 2026-10-06) — https://openrouter.ai/docs/api-reference/list-available-models
2. Vast.ai GPU pricing (pulled 2026-10-06) — https://vast.ai/pricing
3. RunPod pricing, page states "Updated September 27, 2026" — https://www.runpod.io/pricing
4. Ollama library: qwen3.8 — https://ollama.com/library/qwen3.8
5. Ollama library: gemma4 — https://ollama.com/library/gemma4
6. Ollama library: llama4 (Scout 109B/17B active, Maverick 400B; Q4 sizes 67GB/245GB) — https://ollama.com/library/llama4
7. Ollama library: gpt-oss — https://ollama.com/library/gpt-oss
8. llama.cpp — https://github.com/ggml-org/llama.cpp
9. vLLM documentation — https://docs.vllm.ai
10. Artificial Analysis — https://artificialanalysis.ai
11. NVIDIA RTX 4090 specifications (450W TGP) — https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/
12. AnyAPI blog: Why Cost per Million Tokens Is the Wrong Metric — https://anyapi.ai/blog/why-cost-per-million-tokens-is-the-wrong-metric-for-scaling-llms
13. AnyAPI documentation — https://docs.anyapi.ai