What Is a GPU Cluster?
A GPU cluster is a group of servers, each holding several GPUs, wired together so a scheduler can treat them as one pool. The unit you rent or buy is the node. A node is usually 8 GPUs on one baseboard, connected to each other by NVLink and NVSwitch.
Nodes connect to each other over a second network. On NVIDIA systems that is InfiniBand or RDMA Ethernet through ConnectX adapters. Training jobs need this fabric because every step synchronizes gradients across all GPUs. Inference jobs need it only when one model replica is too large for a single node.
Providers sell clusters with an orchestration layer on top. Together GPU Clusters, for example, offers H100, H200, B200, and GB200 nodes with InfiniBand, managed Kubernetes, and Slurm on Kubernetes for training jobs.
GPU Cluster Node Specs: H100 vs H200 vs B200
Memory capacity and bandwidth decide how large a model one node can hold and how fast it decodes. These figures come from NVIDIA's product pages. B200 per-GPU values divide the DGX B200 totals by 8.
| Per 8-GPU node | H100 SXM | H200 SXM | DGX B200 |
|---|---|---|---|
| HBM per GPU | 80GB HBM3 | 141GB | 180GB HBM3e |
| HBM per node | 640GB | 1,128GB | 1,440GB |
| Memory bandwidth per GPU | 3.35TB/s | 4.8TB/s | 8TB/s (64TB/s per node) |
| NVLink | 900GB/s per GPU | Not in quick specs | 14.4TB/s aggregate |
| Node networking | Partner systems vary | Partner systems vary | 8x ConnectX-7, up to 400Gb/s each |
Per-GPU deep dives: NVIDIA H100, NVIDIA H200, and NVIDIA B200.
Sizing a GPU Cluster for LLM Inference
Start with weight memory. Multiply parameter count by bytes per parameter. FP8 uses 1 byte, BF16 uses 2. A 744B model such as GLM-5.3 needs about 744GB in FP8 and about 1,488GB in BF16 before any KV cache.
| 744B model weights | 8x H100 (640GB) | 8x H200 (1,128GB) | DGX B200 (1,440GB) |
|---|---|---|---|
| FP8 (~744GB) | Does not fit, needs 2 nodes | Fits, ~384GB left for KV cache | Fits, ~696GB left for KV cache |
| BF16 (~1,488GB) | Needs 3 nodes | Needs 2 nodes | Needs 2 nodes |
KV cache grows with context length and with the number of requests in flight. The memory left after weights sets how many long-context agent sessions one replica can hold at once. Spanning a replica across nodes adds inter-node traffic to every decode step, so a replica that fits in one node is usually faster than one split over two.
Throughput scales by adding replicas, each a full copy of the model. A cluster serving a large model at high concurrency is several single-node replicas behind a load balancer, rather than one giant job.
GPU Cluster Networking: NVLink and InfiniBand
NVLink handles traffic between GPUs in a node. An H100 SXM has 900GB/s of NVLink bandwidth, against 128GB/s for PCIe Gen5. A DGX B200 reaches 14.4TB/s aggregate across its two NVSwitch chips.
Between nodes, a DGX B200 has 4 OSFP ports serving 8 single-port ConnectX-7 adapters, each up to 400Gb/s InfiniBand or Ethernet. That is roughly 50GB/s per adapter, far below NVLink, which is why tensor parallelism stays inside a node and pipeline or data parallelism crosses nodes.
GPU Cluster vs Dedicated Endpoint
A rented cluster gives you bare nodes. You pick the serving engine, tune kernels, handle failover, and pay for every GPU-hour whether traffic shows up or not. That is the right buy for training, fine-tuning, and research.
If the goal is a model served at a known throughput, a dedicated inference endpoint is the smaller unit. The provider runs the engine and capacity, and you get an OpenAI-compatible URL. Estimate the reserved capacity you need with the dedicated inference calculator, or read the LLM inference guide for serverless versus dedicated tradeoffs.
Frequently Asked Questions
What is a GPU cluster?
A GPU cluster is a group of GPU servers (nodes) connected by a high-bandwidth network so one job can run across all of them. GPUs inside a node communicate over NVLink, and nodes communicate over InfiniBand or RDMA Ethernet.
How many GPUs are in a GPU cluster node?
Most data center nodes hold 8 GPUs. NVIDIA DGX H100 and DGX B200 systems have 8 GPUs each, and HGX H100 and HGX H200 partner systems come with 4 or 8 GPUs.
How much memory does an 8x H100 node have?
An 8x H100 SXM node has 640GB of HBM3 (8 x 80GB). An 8x H200 node has 1,128GB (8 x 141GB). A DGX B200 has 1,440GB of HBM3e.
How big a GPU cluster do I need to serve a 744B model?
In FP8, 744B parameters take about 744GB for weights. That exceeds one 8x H100 node (640GB), so you need two H100 nodes, or one H200 or B200 node. In BF16 the weights take about 1,488GB, which exceeds even one DGX B200. KV cache for concurrent requests comes on top of the weights.
What network does a GPU cluster use?
Inside a node, NVLink connects the GPUs: 900GB/s per H100 SXM GPU and 14.4TB/s aggregate across a DGX B200. Between nodes, a DGX B200 has 8 ConnectX-7 adapters at up to 400Gb/s each over InfiniBand or Ethernet.
Should I rent a GPU cluster or use a dedicated inference endpoint?
Rent a cluster when you are training, fine-tuning, or running your own serving stack and want root access. Use a dedicated endpoint when you only need a model served at a predictable throughput, since the provider handles the serving engine, scaling, and kernel tuning.
The fastest endpoints are private deployments
Morph's top speeds come from dedicated deployments, not shared public endpoints: speculators trained on your traffic, caching tuned to your workload, and volume discounts over public per-token rates. Over 500 billion tokens per day run this way.
Serve open models without running a cluster
Morph serves GLM-5.3, Kimi K3, and DeepSeek V4 Flash on custom kernels behind one OpenAI-compatible API.