--- title: "Morph: Enable Faster Coding Agents" url: "https://www.morphllm.com/blog/faster-agents" description: "A deep dive into how we get morph to be so fast." date: "2025-05-04" author: "Tejas Bhakta" --- # Morph: Enable Faster Coding Agents # Why speed matters Morph is a specialized LLM trained to apply changes to code and files. Same job as Cursor's Composer and Apply. We built ours around speed and accuracy, with a model that does nothing but merge edits. Two numbers we care about: 10,500+ tokens/s, and that speed holds steady on files up to 1500 lines. Applying an edit isn't an unsolved ML problem. GPT-4o and Claude Sonnet can both do it, given the right prompt and enough yelling. The hard part is doing it fast, cheap, and at scale. Frontier models are too slow and too expensive to sit in the apply loop of a coding agent, and without a fast apply step the whole agent feels sluggish. | GPU | Memory (GB) | Memory bandwidth (GB/s) | FP16 Tensor Core (TFLOP/s) | |-----|-------------|------------------------|---------------------------| | A10 | 24 | 600 | 125 | | A100 PCIe/SXM | 80 | 1935/2039 | 312 | | L4 | 24 | 300 | 121 | | L40S | 48 | 864 | 362 | | H100 PCIe/SXM | 80 | 2000/3350 | 756/989 | | H200 SXM | 141 | 4800 | 989 | # What makes apply fast Apply lands 5-8x faster than running the same edit through a general model, and results come back near-instant. Three things about the task make that possible. It's narrow, so we can tune the whole pipeline for one shape of work. The input and output are mostly the same tokens, so we can speculate our way through the unchanged parts. And input and output tend to be similar in length, which changes how we schedule. # The naive version Write the obvious version first and you get this: 1. Load model into GPU memory 2. For each request: tokenize input, run inference, generate tokens, detokenize output 3. Return modified code Fine for a demo. It falls apart under real traffic. The GPU sits idle between requests, requests queue up one behind the other, a fixed batch size wastes parallelism, and every request reserves a whole GPU it doesn't need. At useful model sizes on an A100 or H100 this tops out around 300-400 tokens/second and cost scales linearly with volume. Nobody ships a service on that. # Making it fast The inference pipeline is where most of the work went, and it's the hardest part of making Morph good. A free tier only works if requests feel instant and nobody sits waiting on a batch to fill. Batching is what buys throughput. Run many requests through the model at once and you get far more tokens per second than serving them one at a time. The catch is latency: a request that could have gone straight through now waits for its batch. ## Continuous batching with dynamic SLAs Fixed batch sizes are the wrong tool. We size batches from live traffic and client priority instead. Enterprise requests go through right away; free-tier requests fill batches for throughput. The batch never stalls waiting to be full. ## Speculative decoding We feed the original code back in as the speculation for the decoder. Most edits keep most of the file, so the decoder verifies the unchanged spans cheaply and spends its compute on the parts that actually change. A few things had to be true for that to pay off. Dequantization and attention have to fuse into one pass, or the latency eats the gain. We run custom kernels tuned for the E5M2 format (still in progress), scale adaptively off attention-head statistics, and evict cache by access pattern and priority. On our test suite this costs almost nothing in quality and buys a 2.3x speedup. There's more to it than that, and we'll spare you the rest. Contact us if you want the details. The kernels that fuse dequantization straight into attention, dropping the separate dequant pass, are showing another 15-20% in early benchmarks. ## Custom CUDA kernels (in progress) We're writing GPU operations specifically for code editing. The task is narrow enough that specializing pays: attention ops that cut memory transfers, batch-aware implementations that keep the GPU busy, FlashAttention variants tuned for our workload. OK CUDA is hard. Good CUDA is harder. We're on it. # Building a fast coding agent on top of this Two moves. Apply changes in the background before the user clicks apply, so business users pay the cheaper batched rate. When someone applies directly, process it immediately. # What's next ## Adaptive speculation The speculation strategy is fixed today. We're building one that adjusts to the kind of edit. Type-hint additions are localized, so they can speculate aggressively. Refactors are messier and need to be more conservative. Formatting changes can run flat-out. ## Near-symmetric inference Apply is a symmetric task. It's prefill-heavy, but input and output usually run close in length rather than leaning hard on prefill. That symmetry unlocks a set of optimizations that only make sense when the two sides are balanced. ## Bigger files We've held performance steady up to 2000 lines and want to push further. The wall here is the cost of training and inference, not the approach.