An AI training platform built around eight H100 GPUs lands at roughly 2 TB of system DDR5 per node, compared with the 512 GB per socket (1 TB total) typical of a high-volume dual-socket general-purpose server [S1].
On top of that system DRAM, each H100 brings 80 GB of HBM3, which adds another 640 GB of on-package memory dedicated to the GPU complex on the same board, so a fully populated AI node crosses roughly 2.6 TB of total addressable memory once HBM is counted [S1].
What the capacity gap looks like in plain numbers
The 1 TB figure for a conventional server assumes 512 GB per socket across two sockets using current DDR5 modules, and the source flags that 1 TB as nearly 40% of the total system cost at large-volume pricing of around $10,424 for the full BOM with channel margin included [S1]. Stepping up to a DGX H100, system DDR5 alone grows to 2 TB, while the GPU+HBM complex dominates the bill of materials: the DGX H100 retails for roughly $270,000 per system and Nvidia's gross profit on that system sits close to $190,000 before operating expenses, reflecting how much of the BOM is GPU die, CoWoS packaging, and HBM stacks [S1].
Two consequences follow. First, DRAM as a share of total server cost collapses from roughly 40% in a CPU server to around 2.9% in a DGX H100, and even in an unrealistic scenario where every one of six HBM stacks were replaced with DDR5, DRAM would only climb to 7.15% of system cost [S1]. Second, the absolute DRAM count per node still doubles, because training and inference schedulers assume a 2 TB DDR5 footprint for caching checkpoints, embedding tables, and KV-cache offload, not just the on-package HBM that the GPUs read at full bandwidth [S1].
HBM versus system DRAM, and why both are needed
Modern GPU training demands memory bandwidth that no DDR5 channel array can deliver, which is why HBM3 sits on the GPU package itself; the eight-GPU DGX H100 board therefore carries 640 GB of HBM3 total alongside 2 TB of DDR5, a stack-up that is the opposite of the CPU-server pattern where DRAM is the only high-bandwidth tier [S1].
Industry reporting on AI data-center power reinforces the same hierarchy: modern GPUs draw 700-1,200 W per chip, versus 150-200 W for traditional CPUs, and AI racks run 50-150 kW versus 10-15 kW for conventional racks, so the memory subsystem must feed a much hotter compute pipeline and absorb far larger checkpoint traffic [S2]. That is why the question "how much DRAM does an AI server use" never has a single answer; the system-DRAM count is the figure procurement and capacity planners care about, while HBM is the figure the GPU silicon actually consumes, and both scale together as a node goes from 8-GPU to 16-GPU or 72-GPU rack-scale designs [S1][S2].
Comparison: general CPU server vs AI training node
Four decision criteria line the two architectures up cleanly. (1) System DRAM per node: roughly 1 TB DDR5 on a general server versus about 2 TB DDR5 on a DGX H100, a 2x delta. (2) On-package high-bandwidth memory: 0 GB on a CPU server versus 640 GB of HBM3 across eight GPUs on a DGX H100. (3) DRAM as a share of system cost: about 40% on a CPU server versus 2.9% baseline (7.15% in the "all stacks DRAM" stress case) on a DGX H100, because GPU die, CoWoS, and HBM absorb most of the BOM [S1]. (4) Per-rack power envelope: 10-15 kW for a conventional rack versus 50-150 kW for an AI rack, with individual GPUs at 700-1,200 W versus 150-200 W per CPU [S2].
Where AI servers do not need the extra DRAM
The 2 TB DDR5 figure is a training and large-model inference target, not a universal AI-server rule. Inference-only nodes that run small or distilled models can sit closer to a general server's memory profile, especially when the model fits inside a single GPU's HBM and system DRAM is only used for input pipelines and KV-cache spillover; hardware guidance for inference workloads explicitly calls for "less power and need low latency and high throughput, often relying on smaller GPUs" rather than the eight-GPU training layout [S3].
Edge AI deployments follow the same trim. Where a factory-floor or retail inference box is sized for a single accelerator and a few cameras, the DRAM count tracks a high-end workstation more than a hyperscale AI node, which is one reason operators are pushing inference back on-prem rather than shipping every frame to a hyperscale region [S3].
Why the cost share collapses even when bytes double
The reason a 2x jump in DDR5 capacity translates into a much smaller jump in cost share is that the rest of the AI BOM grew far more. A DGX H100 at roughly $270,000 lists a GPU complex, NVSwitch baseboard, power delivery, and assembly that simply do not exist in a $10,000 CPU box, so even doubling DDR5 capacity barely moves the percentage [S1]. For a buyer, the right planning question is not "is DRAM more expensive on AI servers" but "how much system DDR5 do I need to provision per node, and how many HBM-equipped GPUs do I need to feed it", because memory bandwidth, not capacity alone, sets the training throughput ceiling [S1][S2].
Track these two signals over the next two quarters: HBM revenue concentration across the three suppliers, where the HBM market share race in 2026 sets HBM cost-per-GB, and the DDR4 versus DDR5 contract spread in Q3 2026, which sets whether the 2 TB DDR5 footprint on a new AI node gets cheaper or more expensive into 2027.