Self-Hosted LLM: Hardware, VRAM Math, and When It Beats an API (2026)

What Is a Self-Hosted LLM?

Self-hosting means the model weights sit on machines you operate. That can be a laptop running Ollama, a rack of your own GPU servers, or rented cloud GPUs where you install the serving engine yourself. The model is the same open-weight checkpoint a provider would serve. The difference is who carries the pager.

Teams self-host for three reasons. Some data cannot leave the network under contract or regulation. Some workloads run hard enough, around the clock, that owning GPU-hours costs less than paying per token. And some teams need a fine-tuned checkpoint that no public API serves.

Self-Hosted LLM Hardware: How Much GPU Memory You Need

Weights come first. Multiply parameters by bytes per parameter (2 for BF16, 1 for FP8, 0.5 for FP4), or read the file size off the model card. Then divide by the memory per GPU. The table uses the file sizes published on Hugging Face and gives the floor for weights alone.

Those counts are the minimum, with nothing left over. The KV cache grows with context length and with every concurrent request, so production deployments add GPUs on top. An 8-GPU DGX B200 has 1,440 GB in total. GLM-5.3 fits on one with about 684 GB spare for KV cache. Kimi K3 does not fit on one node at all, and an 8-GPU H200 server (1,128 GB) cannot hold it either.

Precision is a choice only for some models. GLM-5.3 at BF16 would need about 1.5 TB, twice its FP8 release. Kimi K3 was trained quantization-aware with MXFP4 weights, so its 4-bit checkpoint is the reference, not a compressed copy. To size the whole cluster, including networking and replicas, use the GPU cluster sizing guide.

Self-Hosted LLM Software: Ollama, vLLM, and SGLang

Ollama is the shortest path on a single machine. It pulls a model, runs it, and exposes an OpenAI-compatible endpoint on port 11434, so existing OpenAI SDK code only needs a new base URL. This is the client from Ollama's own docs:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the SDK, ignored by Ollama
)

On GPU servers, vLLM and SGLang are the usual engines. vLLM ships an OpenAI-compatible server, so the same client works against a cluster. They batch concurrent requests onto the GPU, which is where throughput comes from once more than one user is sending prompts. The trade-offs between engines are covered in vLLM vs Ollama and SGLang vs vLLM.

Self-Hosted LLM vs API vs Dedicated Deployment

A dedicated deployment sits between the two. The GPUs are reserved for one customer, which covers most of the isolation reasons teams give for self-hosting, while the provider keeps the serving engine, kernels, and on-call. See dedicated inference endpoints for how Morph runs them.

When Self-Hosting an LLM Pays Off

A GPU bills every hour of the month. The per-token API bills only for tokens. So the break-even depends on utilization more than on list prices. Take GLM-5.3 on the 5-B200 floor from the table above. At $$9.98 per B200-hour, five GPUs for a 730-hour month come to $$36,427. Divide that by the tokens you would actually push through them each month, and compare the result with the per-token price you pay today.

Steady traffic keeps the GPUs busy and drives the effective per-token cost down. A workload that runs eight hours a day pays for sixteen idle ones. The dedicated inference calculator runs this comparison for your token volume.

Private LLM Deployment and On-Premise LLMs

An on-premise LLM is the strictest form of self-hosting. You buy the servers and run them in your own data center, so the weights and every prompt stay on hardware you own. A private LLM deployment is the broader term, and it usually means the same model running on single-tenant GPUs in a cloud account or a provider's isolated capacity. Both rely on open-weight models such as GLM-5.3, Kimi K3, and DeepSeek V4 Flash, because closed models do not release weights.

Frequently Asked Questions

What is a self-hosted LLM?

An open-weight language model running on GPUs you control, on your own hardware or rented servers, instead of a provider's API. Prompts and outputs stay inside your network.

How much GPU memory do I need to self-host an LLM?

At least the size of the weights, plus KV cache. GLM-5.3 needs 755.6 GB, which is 6 H200s or 5 B200s at minimum. DeepSeek V4 Flash needs 159.6 GB, or 2 H200s. Kimi K3 needs 1,561 GB, more than one DGX B200 holds.

Is self-hosting an LLM cheaper than an API?

Only at high, steady utilization. Every GPU-hour is billed whether it serves tokens or not. Spiky traffic is cheaper per token on an API.

What software do I use to self-host an LLM?

Ollama on a laptop or workstation, and vLLM or SGLang on GPU servers. Ollama and vLLM both expose OpenAI-compatible endpoints.

What is the difference between self-hosting and a dedicated deployment?

Self-hosting means you run the GPUs and the serving stack. A dedicated deployment reserves single-tenant GPUs for you while the provider runs the stack. Morph bills dedicated B200 capacity at $$9.98 per GPU-hour.