llama.cpp vs Ollama: Raw Performance vs Developer Experience for Local LLMs

Quick Verdict

Architecture: Engine vs Wrapper

Understanding the relationship between these two tools is the key to choosing between them. llama.cpp is the engine. Ollama is the car built around it.

Ollama calls llama.cpp functions like llama_server_init and llama_server_completion through its CGo bridge. Every token generated still passes through the same C++ code path. The overhead comes from the layers above: Go's garbage collector, HTTP request parsing, JSON serialization between the Go server and the C++ engine, and default parameter choices that favor compatibility over raw speed.

Performance Benchmarks

The performance gap is consistent across hardware and model sizes. It grows wider under concurrent load.

Under Concurrent Load

Single-user benchmarks hide the real difference. When multiple users hit the same server, llama.cpp's lower memory footprint and direct hardware access keep it stable. Ollama's additional memory overhead causes earlier VRAM spillover to CPU, which halves generation speed.

At 5 parallel requests, Ollama offloads 38% of computation to CPU because VRAM runs out sooner. llama.cpp's token caching and tighter memory management keep more layers on GPU. This is where the 15-30% single-user gap becomes a 3x throughput gap under production-like conditions.

Feature Comparison

Setup and Workflow

The setup difference is the single biggest factor in adoption. Ollama turns a 30-minute build process into a 30-second install.

# Clone and build with GPU support
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# CUDA (NVIDIA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# Metal (Apple Silicon)
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j

# Download a model (manual)
# huggingface-cli download TheBloke/Llama-3-8B-GGUF \
#   llama-3-8b.Q4_K_M.gguf

# Run inference
./build/bin/llama-server \
  -m ./models/llama-3-8b.Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 4 \
  --port 8080

# Install
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run a model
ollama pull llama3.2
ollama run llama3.2

# Or start the API server
ollama serve

# That's it. API available at:
# http://localhost:11434/v1/chat/completions

# Custom configuration (Modelfile)
# FROM llama3.2
# PARAMETER num_ctx 32768
# PARAMETER temperature 0.7
# SYSTEM "You are a helpful assistant."

# ollama create my-model -f Modelfile
# ollama run my-model

With llama.cpp, you need cmake, a C++ compiler, and the appropriate GPU SDK (CUDA Toolkit, Xcode for Metal). You download GGUF files from Hugging Face yourself, choose your quantization format, and configure every runtime parameter. This takes 15-30 minutes on a first setup.

With Ollama, you run one install command and one pull command. Ollama downloads pre-quantized models from its registry, detects your GPU, and starts serving. The entire process takes under a minute. The cost: you get Ollama's default quantization choices (usually Q4_0 or Q4_K_M), default context window (2048 tokens), and default parallelism (2 slots).

Quantization and Model Formats

Both tools run GGUF models, the format designed by the llama.cpp project. The difference is how much control you get over the quantization process.

llama.cpp gives you access to every quantization format through its llama-quantize tool. You can quantize your own models from FP16/FP32 source weights, choosing the exact tradeoff between size, speed, and quality. The K-quant formats (Q4_K_M, Q5_K_M) use importance matrices to preserve quality in critical layers while aggressively compressing less important ones.

Ollama pulls pre-quantized models from its registry. You get what the model publisher chose. You can bring your own GGUF files via a Modelfile (FROM ./my-model.gguf), but the quantization step itself still requires llama.cpp's tooling. For most users, the registry defaults (typically Q4_K_M or Q4_0) are good enough. For researchers or teams optimizing for specific hardware, the ability to test Q5_K_M vs Q4_K_S vs IQ4_XS on your exact workload matters.

API and Integration

Both tools expose OpenAI-compatible REST APIs. The integration story is nearly identical for downstream applications.

# Start the server
llama-server \
  -m model.gguf \
  --n-gpu-layers 99 \
  --port 8080

# Use with OpenAI SDK
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="local-model",
    messages=[{"role": "user",
               "content": "Hello"}]
)

# Start the server (auto on install)
ollama serve

# Use with OpenAI SDK
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user",
               "content": "Hello"}]
)

The client code is identical except for the base URL and port. Any tool that supports OpenAI-compatible endpoints works with both: Open WebUI, Continue, Roo Code, LangChain, LlamaIndex, and Morph. Ollama's API also includes its own native REST endpoints at /api/chat and /api/generate for features not covered by the OpenAI spec, like model management and embedding generation.

Hardware Support

llama.cpp has the broadest hardware support of any local inference engine. Ollama inherits most of it, but not all.

The practical difference shows up in edge cases. If you need to split a 70B model across a 12GB GPU and 32GB of system RAM, llama.cpp lets you specify exactly how many layers go to GPU (--n-gpu-layers 28) and how threads are allocated for CPU layers. Ollama handles this automatically, but its VRAM estimation is occasionally off, leading to slower inference when it misjudges the split.

On Apple Silicon specifically, llama.cpp talks directly to Metal for GPU compute. Ollama recently added an MLX backend option in preview, which can outperform the Metal path by 10-20% on M3/M4 chips. Neither tool matches the speed of running MLX natively, but both are within striking distance for most practical workloads.

When to Use llama.cpp

When to Use Ollama

Frequently Asked Questions

Is llama.cpp faster than Ollama?

Yes. Benchmarks consistently show 15-30% higher throughput with llama.cpp on the same hardware. One measurement: 137.79 tok/s vs 122.07 tok/s on DeepSeek R1 1.5B, a 26.8% gap. The difference comes from Go runtime overhead, HTTP serialization in the wrapper layer, and conservative default parameters. Under concurrent load, the gap widens to 3x because Ollama spills to CPU sooner.

Is Ollama just a wrapper around llama.cpp?

Yes. Ollama embeds llama.cpp via CGo bindings. The Go layer adds model registry management (pull, push, list, delete), an HTTP API server, GPU auto-detection, Modelfile configuration, and model lifecycle management. All actual inference happens in the same C++ code. The wrapper adds convenience at the cost of performance overhead and reduced configurability.

Should I use Ollama or llama.cpp for production?

For production with concurrent users, llama.cpp provides better throughput, lower memory overhead, and more tuning options (batch size, parallelism, layer placement, speculative decoding). For single-user development or prototyping, Ollama's setup speed and model management save significant time. A common pattern: develop with Ollama on your laptop, deploy with llama.cpp on your server.

Does Ollama support the OpenAI API format?

Yes. Ollama exposes OpenAI-compatible endpoints at http://localhost:11434/v1/chat/completions. llama.cpp's server provides the same at http://localhost:8080/v1/chat/completions. Both work with the OpenAI Python and JavaScript SDKs. Set base_url to the local address and use any model name you have loaded.

Can I use llama.cpp or Ollama with Morph?

Yes. Morph works with any OpenAI-compatible endpoint. Set the base URL to your local server address, and Morph's fast-apply code editing engine works with your locally-hosted model. This enables fully offline, private development workflows while keeping the same API interface you would use with cloud models.

What about vLLM? When should I use that instead?

vLLM is the right choice for multi-GPU production serving with tensor parallelism and continuous batching. Neither llama.cpp nor Ollama does true tensor parallelism, they split layers across GPUs sequentially. If you have 2+ high-end GPUs and need to serve many concurrent users, vLLM outperforms both. For single-GPU or CPU+GPU hybrid setups, llama.cpp is faster and lighter.