Which AI model (LLM) and server size should I use with Ollama/Lama.cpp/Codex/Claude Code?

Hosting LLMs requires powerful hardware. Preferably you use one or more powerful GPUs, but it is also possible to run LLMs using only CPU and RAM. This is especially interesting when you want to host a model internally, when GPUs are too expensive, or when data is not allowed to be sent to external AI services.

Please note: CPU-only inference is usually considerably slower than GPU inference. As a rough estimate, expect performance to be 10 to 50 times slower, depending on the model, CPU, memory, quantization used and context size. For interactive chat this can be annoying, but for tasks such as code analysis, documentation, batch processing or agentic coding, CPU-only can still be useful if speed is not the most important requirement.

In this article we explain which models and which server size (RAM or RAM + VRAM) you need to effectively use LLMs with Ollama, Llama.cpp, Codex and Claude Code.

The model file is not the same as RAM usage

A common mistake is to look only at the download size of the model. If a model in Ollama is 20 GB in size, for example, that does not mean the full runtime only uses 20 GB RAM.

Total RAM usage roughly consists of:

That is why a 7.2 GB model may sometimes work in llama.cpp on an Ubuntu server with 8 GB RAM, but not in Ollama. Ollama itself uses additional memory and the operating system also needs RAM. At such tight margins, 500 MB to 1 GB of overhead can already make the difference between “works” and “out of memory”.

Quantization: less RAM, slightly lower quality

To reduce hardware requirements, you can use quantized models. Simply put, the model is stored more compactly. You usually sacrifice a little accuracy, but significantly reduce the required RAM or VRAM.

For many local models, Ollama uses a 4-bit quantized variant by default when you only use the model name, for example:

ollama pull qwen3.5

This normally fetches the default tag, for example qwen3.5:latest. If you deliberately want a different quantization, you need to choose a specific tag, for example:

ollama pull qwen3.5:27b
ollama pull qwen3.5:27b-q8_0

The exact tag names differ per model. Therefore always check the model’s “Tags” page in Ollama.

Example: Qwen3.5 and RAM per quantization

The table below gives a practical impression of model size/RAM direction for Qwen3.5-like models. This is not the same as full runtime usage at maximum context, but it helps to understand the sizes.

Context size and K/V cache

The context size determines how many tokens the model can take into account at the same time. For simple chat, 4K to 16K context is often enough. For RAG, long documents and codebases, you’ll sooner want 32K to 64K. For Codex / Claude Code-like workflows, 64K or more is often desirable (preferably even 256K or more), because the agent needs to keep a lot of code, terminal output, tool results and previous steps in context.

But large context is not free. During inference, the model has to store information for every token in the so-called K/V cache.

The simple explanation:

• The model file is the “brain”.
• The prompt and conversation are what you ask at that moment.
• The K/V cache is the working memory that allows the model to keep using that context efficiently.

As a result, a large context can cost tens of GB of additional RAM.

Rough formula for K/V cache

The K/V cache depends on the model’s architecture:

K/V cache ≈ 2 × layers × KV_heads × head_dim × context_tokens × bytes_per_value

The 2 stands for Key and Value. The number of layers, KV heads and head dimensions differs per model. You therefore cannot reliably calculate the K/V cache based only on “20B” or “70B”.

Example: 20B model with 256K context

Suppose you have a modern 20B-like model with Grouped Query Attention, for example:

48 layers
8 KV-heads
128 head-dim
256K context

Then you get approximately:

Important: a Q4 model does not automatically mean the K/V cache is Q4 as well. In many runtimes, the K/V cache is F16 by default unless you explicitly enable K/V cache quantization. This is an important reason why a model with a seemingly acceptable file size can still require much more RAM once you increase the context.

Example: GLM-5.2 with 1M context

GLM-5.2 is interesting because it is aimed at long agentic engineering tasks and offers a context of around 1M tokens. Due to the MLA/KV compression used, the K/V cache can be much more efficient than with classic attention.

With a full 1M context and Q4 K/V cache, you roughly end up around 25 GiB K/V cache. With F16/BF16, that would be closer to around 88 GiB K/V cache.

This is only realistic if the runtime properly supports the model’s compressed K/V cache. If a runtime were to expand the cache to classic K/V heads, 1M context quickly becomes impractical on CPU-only servers.

Ollama versus llama.cpp

Ollama and llama.cpp use similar techniques under the hood, but in practice llama.cpp is often slightly easier to tune if you are at the absolute limit of your RAM.

How large should the server be?

The RAM recommendations below are based on CPU-only servers, Q4 models and normal to reasonably large context usage. They are intentionally practical and not theoretically minimal.

Model suitability for Codex / Claude Code

Codex and Claude Code-like tools place higher demands on a model than ordinary chat. The model must not only generate text, but also:

• be able to understand code;
• analyse multiple files at the same time;
• structure tool calls reliably;
• interpret terminal output;
• recover from errors;
• follow instructions strictly;
• not hallucinate too quickly;
• sustain long tasks across multiple steps.

That is why 2B and 4B models are unsuitable for these kinds of workflows. They can generate simple code examples, but often fail on real repositories, refactors, tests, dependency problems and multi-step tasks.

The star rating below is intended practically: how suitable is the model as a backend for Codex / Claude Code-like agentic coding? The rating is not only about benchmark scores, but also about tool use, coding quality, context and realistic usability on CPU-only servers.

Use cases per server size

If speed is less important (15 tokens per second) and you mainly want it to work well, gpt-oss 120B is one of the most logical choices. The model is relatively compact for its class, is designed for agentic workflows, supports tool use and reasoning, and fits more practically in a server with 96 GB to 128 GB RAM than many other 100B+ models.

Practical settings

1. Keep context deliberately low

Do not automatically use the maximum context. For example, start with:

2. Use K/V cache quantization where useful

If you use long context, K/V cache quantization can save a lot of RAM. A Q4 K/V cache can save tens of GB at large context compared with F16/BF16. Q8 K/V cache is often a safe middle ground. Q4 K/V cache saves more RAM, but comes at a greater cost to quality.

3. Prefer a better 20B–35B model over a weak 70B model

Bigger is not always better. For Codex / Claude Code, a model that is well trained on coding, tool use and instruction following is often more useful than a larger general chat model.

4. Do not use small models for serious agents

2B and 4B models are attractive because they run quickly and cheaply, but for real agentic coding they are disappointing. They often lack reliable tool calling, good error recovery, code understanding, long-term planning, consistent patches and understanding of project structure.

Recommended models

• Best choice for 32 GB RAM: Devstral Small 2 24B or gpt-oss 20B. Devstral is strong for coding workflows and relatively lightweight. gpt-oss 20B is attractive for tool use, reasoning and agentic focus.
• Best choice for 48 GB RAM: Qwen3.6 35B-A3B or DeepSeek-R1 32B. This class is powerful enough for serious coding work and still fits reasonably well on CPU-only servers.
• Best choice for 128 GB RAM: gpt-oss 120B. For a CPU-only server with 128 GB RAM, where speed is less important than quality, gpt-oss 120B is one of the strongest choices (while still performing at 15 tokens per second). It fits more practically in memory than many other large models, is designed for agentic workflows and is strong in tool use and reasoning.
• Best cloud option via Ollama: GLM-5.2 cloud (also available locally, but at least 256GB RAM required). If local hosting is not a hard requirement, a cloud-hosted version of GLM-5.2 is interesting because of its very large context and focus on long-horizon agentic engineering. It is especially relevant for large repositories and long tasks where a lot of context is needed.