How Much Memory Does a Local LLM Need?
Quick Answer
A local LLM needs enough memory for more than its model file. You also need headroom for the inference runtime, KV cache, operating system and concurrent requests. For that reason, a model that technically fits in 32GB can still be a poor experience once context length grows. Memory planning should start with the actual quantized model size, then add headroom for cache and services.
For a compact AI system, 32GB is best treated as a small-model tier, 64GB opens practical room for many quantized 20B–35B-class workloads, and 96–128GB provides substantially more room for larger models, longer context or multiple services. These are planning tiers, not universal model requirements.
Key Takeaways
· Model weights are only the first memory consumer.
· KV cache grows with context length and active sequences.
· Quantization reduces model weight memory but does not remove runtime and cache overhead.
· Unified memory gives the GPU access to a larger shared pool, but the OS and CPU also use that pool.
· More capacity improves what can fit; it does not automatically improve tokens per second.
Why the Model File Size Is Not the Full Requirement
If a quantized model file is 20GB, allocating exactly 20GB of memory is not enough. The runtime needs working memory, kernels and temporary buffers. The operating system needs RAM. An API server may run a tokenizer, Python process, monitoring tools and a web interface. If RAG is involved, embeddings, a vector database and document processing services need additional memory.
This is why local AI hardware should have a safety margin. A system that remains below memory pressure is easier to tune and far less likely to fall back to slower paths or fail with an out-of-memory error.
KV Cache Is the Part That Grows With Context
During autoregressive generation, the model stores key/value states for prior tokens so it does not recompute the complete history for every new token. The Hugging Face KV cache guide explains that cache memory can become a bottleneck for long-context generation. Longer prompts mean more cached states. More simultaneous conversations mean several caches at the same time.
This is exactly what appears in the CWWK Ryzen AI Max+ 395 benchmark. The 35B-A3B model remained highly responsive at short inputs, while throughput dropped and TTFT rose as the prompt moved from hundreds of tokens to several thousand. Capacity and latency therefore need to be planned together.
How Quantization Changes the Calculation
Quantization stores model weights with fewer bits, reducing memory use and often making larger models practical on local hardware. However, it does not shrink every tensor or every runtime allocation by the same ratio. The correct planning input is the actual model artifact you intend to deploy and the behavior of the chosen backend.
Also separate “can load” from “can serve well.” A large model may fit after aggressive quantization but still decode slowly because compute or memory bandwidth is the bottleneck.
What 32GB, 64GB and 128GB Mean in Practice
|
System memory |
Practical direction |
Main caution |
|
32GB |
Small local assistants, development, smaller quantized models |
Limited headroom for large context and multiple services |
|
64GB |
Many quantized 20B–35B-class local inference workloads; local RAG |
Still requires context and concurrency control |
|
96–128GB |
More model residency, larger caches, multi-service or larger quantized models |
Capacity does not equal discrete-GPU performance |
CWWK tested Qwen3.5-27B-AWQ and Qwen3.5-35B-A3B-AWQ on 64GB memory. The current CW-A9 supports up to 128GB LPDDR5X-8000, so the higher-memory configuration mainly provides additional residency and cache headroom rather than a guaranteed multiplier in throughput.
Unified Memory Changes Where the Limit Appears
On a conventional desktop with a discrete GPU, system RAM and GPU VRAM are separate pools. If a model exceeds VRAM, the runtime may have to offload data or split the workload. Ryzen AI Max+ 395 uses a high-capacity unified-memory design, allowing the integrated GPU and CPU to access the same physical memory pool.
That is useful for models that would not fit on a 16GB or 24GB discrete GPU. But there is a tradeoff: the memory is shared with the rest of the system, and its bandwidth and GPU compute characteristics are different from high-end discrete accelerators. Treat unified memory as a capacity advantage, not as a claim that all 128GB behaves like dedicated VRAM.
Leave Memory for the Rest of the AI Stack
· Operating system and desktop environment
· Inference framework and API server
· KV cache at the target context length
· Vector database and embedding model for RAG
· Containers and monitoring
· A second model or reranker, if used
If the system is a shared local AI server, concurrency deserves explicit testing. A single interactive session may be comfortable while eight parallel sessions exhaust the cache budget. Use the vLLM benchmark tools or an equivalent load test with the same prompt sizes your users will actually send.
FAQ
Can a 27B model run in 32GB?
Some quantized 27B models may fit, but practical headroom can be tight. Check the real artifact size, KV cache, runtime and OS memory instead of relying on parameter count alone.
Is 128GB necessary for CW-A9?
No. CWWK benchmarked 27B and 35B AWQ models on 64GB. The 128GB configuration is most valuable when you want more headroom for larger models, longer context, multiple models or additional local AI services.
Will more RAM improve TTFT?
Only if the current system is memory constrained or offloading. If the workload already fits, TTFT is more likely to be limited by prompt processing, bandwidth, compute and software efficiency.
Does context length consume memory even before generation starts?
Yes. Prompt processing builds cache state for the input, and that cache grows as the sequence grows.
Related CWWK Resources
· Local LLM Hardware Requirements
· Ryzen AI Max+ 395 Local LLM Benchmark
· TTFT vs TPS vs ITL: Local LLM Benchmark Metrics
External References
· Hugging Face KV cache strategies
