Local LLM Hardware Requirements: Memory, Compute & Storage

Quick Answer

The most important local LLM hardware requirement is enough usable memory to hold the model, its runtime overhead and the KV cache at the context length and concurrency you actually plan to use. After memory capacity, the next constraints are memory bandwidth, accelerator support, compute throughput, storage speed and software compatibility. CPU core count matters, but for modern accelerated inference it is rarely the only specification that determines tokens per second.

A practical buying order is: choose the model class and quantization, set a realistic context length, estimate concurrency, verify the GPU or accelerator path supported by your framework, leave memory headroom, then size NVMe storage and networking. A compact platform such as the CWWK CW-A9 can be a strong local inference system because it combines Ryzen AI Max+ 395, Radeon 8060S and high-capacity LPDDR5X unified memory, but it still needs to be matched to the workload rather than judged by a single TOPS number.

Key Takeaways

· Memory capacity decides whether the model and cache can stay resident; memory bandwidth and compute decide how quickly tokens are processed.

· Context length and concurrency consume memory even after model weights have loaded.

· GPU acceleration remains the primary path for many local LLM frameworks; an NPU does not automatically replace the GPU for vLLM or PyTorch workloads.

· Fast NVMe matters for model loading, multiple model files, indexes and local datasets, but it does not replace sufficient RAM or accelerator memory.

· Software support is a hardware requirement: driver, kernel, ROCm/CUDA backend and framework compatibility must be checked before purchase.

1. Start With the Model, Not the PC

A hardware plan should start from the model family you intend to run. Parameter count gives only a first approximation. Quantization changes how much space the weights require, and model architecture changes how much compute is active for each token. A dense 27B model and a 35B sparse MoE model can behave very differently even when their names look similar.

2. Memory Has Four Separate Jobs

Local LLM memory planning is often simplified into “model size equals required RAM.” That is incomplete. The system needs memory for at least four things: model weights, runtime workspace, KV cache and the operating system or other services. If the model barely fits, there may be no room for longer context, several simultaneous users, an embedding model, a vector database or a second model.

Model weights

Quantization reduces the space used by model weights, but real files also include metadata, tensors with different precision and framework overhead. Use the actual model file size as the starting point rather than multiplying parameter count by a generic bits-per-weight formula and treating the result as exact.

KV cache

Autoregressive generation reuses attention keys and values for previous tokens. The Hugging Face cache documentation notes that KV cache memory can become a significant bottleneck during long-context generation. As input length and the number of concurrent sequences increase, cache requirements rise. This is one reason a system that loads a model successfully at 4K context can struggle at 32K or 128K.

Runtime and system headroom

Inference engines need temporary buffers and memory pools, while the OS, browser, containers, databases and monitoring stack need their own RAM. On a unified-memory platform, CPU and GPU workloads also draw from the same physical pool. Leave deliberate headroom instead of planning for 100% model occupancy.

3. Memory Bandwidth Matters After the Model Fits

Once capacity is sufficient, token generation often becomes sensitive to how quickly weights and cache data can be moved. AMD specifies Ryzen AI Max+ 395 with a 256-bit LPDDR5X interface and support for LPDDR5X-8000. The current CW-A9 product is offered with up to 128GB of that memory. The combination is important: capacity allows larger models to remain resident, while bandwidth influences how quickly the compute units can be fed.

This is why “128GB unified memory” should not be interpreted as equivalent to a 128GB high-end discrete GPU. The capacity is valuable, but compute throughput, effective memory bandwidth, software kernels and power limits remain different. Fit and speed are separate questions.

4. Choose the Main Compute Path

GPU

For current local LLM serving, GPU acceleration is the most common high-throughput path. Frameworks such as PyTorch and vLLM depend on supported GPU kernels and drivers. Radeon 8060S provides the main general-purpose parallel compute resource in a Ryzen AI Max+ 395 system.

CPU

The CPU still matters for tokenization, request scheduling, decompression, API services, retrieval, databases and any operations that do not remain on the GPU. More cores also help when one system runs an inference server plus RAG, Docker and background services. However, buying a higher-clocked CPU will not fix an unsupported GPU backend or insufficient memory.

NPU

NPUs are useful for supported low-power AI tasks, but they should not be treated as a drop-in replacement for the GPU in mainstream large-model serving. AMD lists up to 50 NPU TOPS and up to 126 combined platform TOPS for Ryzen AI Max+ 395, yet framework support determines whether a particular model can use that NPU at all. Always check the runtime you intend to deploy.

5. Storage: NVMe Affects Loading and the Whole AI Pipeline

Model weights are large files, and local AI systems often store several quantizations, embedding models, datasets and vector indexes. NVMe storage reduces model load time and keeps data ingestion from becoming an avoidable bottleneck. A 1TB drive can become tight quickly once multiple 20–60GB model files, caches, containers and datasets are stored locally.

6. Network Requirements Depend on Where the Clients Are

If the model is used only on the same desktop, Ethernet speed has little effect on token generation. Networking becomes important when the machine acts as a local inference server for other PCs, a team, a NAS-backed RAG system or several services. Dual 2.5GbE is enough for most text inference and API traffic, because token streams are tiny compared with storage traffic.

7. Software Compatibility Is Part of the Hardware Specification

An accelerator is only useful when the selected OS, driver, framework and model format support it. AMD maintains current ROCm documentation for Radeon and Ryzen and specific Strix Halo optimization guidance. Kernel and ROCm versions can materially affect whether GPU compute initializes correctly and whether memory is exposed as expected.

8. Practical Local LLM Hardware Profiles

These are planning directions, not minimum specifications for every model. Actual memory depends on quantization, architecture, context, batch size and runtime. The purpose is to identify the dominant constraint before buying hardware.

9. Where CW-A9 Fits

CW-A9 sits between a conventional mini PC and a discrete-GPU workstation. Its strength is a compact x86 platform with 16 Zen 5 CPU cores, Radeon 8060S graphics, up to 128GB LPDDR5X-8000, dual NVMe and dual 2.5GbE. That makes it well suited to local inference, private assistants, development, RAG and compact multi-service AI workstations where a large discrete GPU chassis is undesirable.

10. A Better Way to Size Local AI Hardware

1. Choose the exact model family and quantization you intend to run.

2. Set a realistic context length; do not size around the model’s theoretical maximum unless you truly need it.

3. Define concurrency: one local user, several local tools, or a shared API server.

4. Check the supported compute backend for your OS and framework.

5. Add memory headroom for KV cache, runtime and other services.

6. Plan enough NVMe for model files, indexes, containers and datasets.

7. Add faster networking only when data is remote or the server is shared.

8. Benchmark the final stack using TTFT, ITL and throughput rather than relying on TOPS alone.

FAQ

Is 32GB enough for local LLMs?

It can be enough for smaller or more aggressively quantized models, but it provides limited headroom for 27B–35B models, long context or multiple services. Choose memory from the actual model file and runtime requirements.

Is 64GB enough for Qwen3.5-27B or 35B?

CWWK tested AWQ versions of both on a 64GB Ryzen AI Max+ 395 system. That proves those tested configurations can run, but context length, concurrency and software stack still determine practical performance.

Does more RAM make tokens per second faster?

Not necessarily. More memory can prevent offload and OOM conditions and allow larger caches, but once the workload fits, bandwidth and compute often become the main limits.

Do I need 10GbE for a local LLM server?

Usually not for text generation alone. 10GbE is useful when the inference node moves large datasets, images or video to and from separate storage or serves a high-throughput data pipeline.

Related CWWK Resources

· Ryzen AI Max+ 395 Local LLM Benchmark

· Local LLM Memory Requirements

· CW-A9 for Local AI Workloads

External References

· AMD Ryzen AI Max+ 395 specifications

· AMD ROCm on Radeon and Ryzen

· AMD Strix Halo ROCm optimization guidance

· Hugging Face KV cache documentation

· vLLM benchmark metric definitions