Installing and using llama.cpp on Linux, macOS and Windows

llama.cpp is an open-source project that lets you run Large Language Models locally via C/C++. The project is especially popular for GGUF models: model files suitable for fast and relatively lightweight inference on CPUs and GPUs. 

Llama.cpp is used as an inference backend by many popular AI apps such as Ollama and LM Studio, but uses less hardware and performs faster when you use it without Ollama or LM Studio. 

In this guide, you install llama.cpp on Linux, macOS or Windows. Linux is the most relevant option for a server without a GPU (for example a VPS). After that, you run your first prompt, start a local API server and test it with curl. On a server without a GPU, use small or quantized models; large models are slow or do not fit in the available RAM.

Before you start

For this guide you need the following:

• Linux: a server with Ubuntu, Debian, AlmaLinux, Rocky Linux or CentOS Stream.
• macOS: a Mac with Homebrew or the option to install Homebrew.
• Windows: Windows 10 or 11 with Developer PowerShell for VS 2022.
• At least 4GB RAM for small test models. Use 8GB or more if you want to test larger models more comfortably; see this guide on what size of server and what LLM to pick.

GGUF is the model file format used by llama.cpp and supports quantization. A quantized model is a smaller model file where the weights are stored more compactly. As a result, the model needs less RAM and disk space, which is especially important on a VPS without a GPU.

OpenBLAS is mainly relevant for CPU-only use. It helps with prompt processing using larger batch sizes, but does not make token generation faster. On macOS, llama.cpp uses the Accelerate framework for BLAS and Metal for GPU acceleration by default.

Installing llama.cpp on Linux

Use these steps for a Linux server. The CPU build with OpenBLAS is the best starting point for a VPS without a GPU. The CUDA build is only relevant when your server has an NVIDIA GPU and the NVIDIA driver plus CUDA toolkit are already present.

Step 1

Connect to your server via SSH, the VPS console (VPS), or the OpenStack console (OpenStack instance).

Step 2

First update your server so the latest software packages are available.

Ubuntu / Debian:

sudo apt -y update && sudo apt -y upgrade

Check whether a restart is required and, if so, reboot your server first:

test -f /var/run/reboot-required && echo "Restart required" || echo "No restart requirement reported"

AlmaLinux / Rocky Linux / CentOS Stream:

sudo dnf -y update

Check whether a restart is required and, if so, reboot your server first:

sudo dnf install -y dnf-utils
sudo needs-restarting -r

Step 3

Install the build dependencies. Git downloads the source code, CMake creates the build configuration and the compiler builds the llama.cpp tools. The OpenBLAS package is used for CPU acceleration during prompt processing.

Ubuntu / Debian:

sudo apt -y install git build-essential cmake curl libcurl4-openssl-dev libssl-dev libopenblas-dev

AlmaLinux / Rocky Linux / CentOS Stream:

sudo dnf -y install git gcc gcc-c++ make cmake curl libcurl-devel openssl-devel openblas-devel

The libcurl package is needed if you want to download models directly from Hugging Face using the -hf option. The OpenSSL package is needed for HTTPS/TLS support. If openblas-devel is not available on your Linux distribution, use the fallback build without OpenBLAS in step five.

Step 4

Download the llama.cpp source code:

cd ~
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Step 5

Compile (build) llama.cpp:

CPU build with OpenBLAS:

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build --config Release -j "$(nproc)"

Fallback CPU build without OpenBLAS:

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j "$(nproc)"

NVIDIA GPU build with CUDA:

First check whether the NVIDIA driver and CUDA toolkit are available:

nvidia-smi
nvcc --version

If both commands show version information, build llama.cpp with CUDA:

cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build-cuda --config Release -j "$(nproc)"

The CUDA build uses build-cuda instead of build. When running llama-cli or llama-server, add the -ngl 99 option to offload model layers to the GPU. If GPU offloading fails or VRAM is full, use a lower number.

Step 6

Check whether llama-cli and llama-server were built correctly:

CPU build:

./build/bin/llama-cli --help
./build/bin/llama-server --help

CUDA build:

./build-cuda/bin/llama-cli --help
./build-cuda/bin/llama-server --help

If you see the help text for both programmes, the installation was successful.

Step 7

Finally, add the llama.cpp programmes to your PATH. After that, you can run llama-cli and llama-server from any directory.

If you use the CPU build, run:

mkdir -p ~/.local/bin
ln -sf "$HOME/llama.cpp/build/bin/llama-cli" "$HOME/.local/bin/llama-cli"
ln -sf "$HOME/llama.cpp/build/bin/llama-server" "$HOME/.local/bin/llama-server"

If you use the CUDA build, run this instead:

mkdir -p ~/.local/bin
ln -sf "$HOME/llama.cpp/build-cuda/bin/llama-cli" "$HOME/.local/bin/llama-cli"
ln -sf "$HOME/llama.cpp/build-cuda/bin/llama-server" "$HOME/.local/bin/llama-server"

Then check whether ~/.local/bin is in your PATH and, if not, correct it immediately:

echo "$PATH" | grep -q "$HOME/.local/bin" || echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.profile export PATH="$HOME/.local/bin:$PATH"

Finally, test whether the commands are available from any directory:

cd ~
llama-cli --help
llama-server --help

Open a new SSH session if llama-cli or llama-server is not found immediately.

Installing llama.cpp on macOS

On macOS, llama.cpp uses the Accelerate framework for BLAS and Metal for GPU acceleration on supported Apple hardware by default. You therefore do not need to install OpenBLAS separately.

Step 1

Check whether Homebrew is available:

brew --version

If Homebrew is not installed yet, install it using the official installer script:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 2

Install Git and CMake:

brew install git cmake

Step 3

Download and compile (build) llama.cpp:

cd ~
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j "$(sysctl -n hw.logicalcpu)"

If you explicitly want to build without Metal, add -DGGML_METAL=OFF to the CMake command.

Step 4

Check whether llama-cli and llama-server were built:

./build/bin/llama-cli --help
./build/bin/llama-server --help

Step 5

Add the llama.cpp programmes to your PATH. After that, you can run llama-cli and llama-server from any directory.

To do this, create symbolic links in the Homebrew bin directory:

ln -sf "$HOME/llama.cpp/build/bin/llama-cli" "$(brew --prefix)/bin/llama-cli"
ln -sf "$HOME/llama.cpp/build/bin/llama-server" "$(brew --prefix)/bin/llama-server"

Then test whether the commands are available from any directory:

cd ~
llama-cli --help
llama-server --help

Are the commands not found? Close your terminal window, open a new terminal window and try again.

Installing llama.cpp on Windows

On Windows, use the Visual Studio Build Tools. OpenBLAS on Windows requires additional manual library configuration. For a simple installation, therefore use the default CPU build, or the CUDA build if you use an NVIDIA GPU.

Step 1

Open PowerShell as administrator and install Git, CMake and Visual Studio 2022 Build Tools:

winget install --id Git.Git -e
winget install --id Kitware.CMake -e
winget install --id Microsoft.VisualStudio.2022.BuildTools -e --override "--wait --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"

If you see a window indicating that an update is available, install the update as well. Close PowerShell after the installation.

Step 2

Open ‘Developer PowerShell for VS 2022’ from the Start menu. Use this window for the rest of the Windows steps.

Step 3

Download the llama.cpp source code:

cd $env:USERPROFILE
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Step 4

cmake -B build -DLLAMA_BUILD_BORINGSSL=ON
cmake --build build --config Release 

nvidia-smi
nvcc --version

cmake -B build-cuda -DGGML_CUDA=ON -DLLAMA_BUILD_BORINGSSL=ON
cmake --build build-cuda --config Release

Step 5

Check whether llama-cli and llama-server were built correctly:

CPU build:

.\build\bin\Release\llama-cli.exe --help
.\build\bin\Release\llama-server.exe --help

CUDA build:

.\build-cuda\bin\Release\llama-cli.exe --help
.\build-cuda\bin\Release\llama-server.exe --help

Step 6

Add the llama.cpp programmes to your user PATH. After that, you can run llama-cli.exe and llama-server.exe from any directory.

If you use the CPU build, run this in Developer PowerShell for VS 2022:

$llamaBin = "$env:USERPROFILE\llama.cpp\build\bin\Release"
$userPath = [Environment]::GetEnvironmentVariable("Path", "User") 

if (($userPath -split ';') -notcontains $llamaBin) { $newPath = if ([string]::IsNullOrWhiteSpace($userPath)) { $llamaBin } else { "$userPath;$llamaBin" } [Environment]::SetEnvironmentVariable("Path", $newPath, "User") }

$env:Path = "$env:Path;$llamaBin"

If you use the CUDA build, run this instead:

$llamaBin = "$env:USERPROFILE\llama.cpp\build-cuda\bin\Release"
$userPath = [Environment]::GetEnvironmentVariable("Path", "User")

if (($userPath -split ';') -notcontains $llamaBin) { $newPath = if ([string]::IsNullOrWhiteSpace($userPath)) { $llamaBin } else { "$userPath;$llamaBin" } [Environment]::SetEnvironmentVariable("Path", $newPath, "User") }

$env:Path = "$env:Path;$llamaBin"

Then test whether the commands are available from any directory:

cd $env:USERPROFILE
llama-cli.exe --help
llama-server.exe --help

Is llama-cli.exe or llama-server.exe still not found immediately? Open a new PowerShell window.

Running your first prompt

In this example, llama.cpp automatically downloads a small GGUF model from Hugging Face. The model is then used to answer a short prompt.

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -p "Explain in three sentences what llama.cpp is." -n 128

llama-cli.exe -hf ggml-org/gemma-3-1b-it-GGUF -p "Explain in three sentences what llama.cpp is." -n 128

./build-cuda/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -p "Explain in three sentences what llama.cpp is." -n 128 -ngl 99

.\build-cuda\bin\Release\llama-cli.exe -hf ggml-org/gemma-3-1b-it-GGUF -p "Explain in three sentences what llama.cpp is." -n 128 -ngl 99

Interactive chatting

Start llama-cli without a fixed prompt to chat interactively.

Linux / macOS:

cd ~/llama.cpp
./build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -cnv

Windows:

cd $env:USERPROFILE\llama.cpp
.\build\bin\Release\llama-cli.exe -hf ggml-org/gemma-3-1b-it-GGUF -cnv

The -cnv option activates conversation mode. Models with a built-in chat template use that template automatically. Stop the interactive session with ctrl + c.

Starting a local API server

llama-server starts a local HTTP server. This server has endpoints similar to the OpenAI Chat Completions API, making it easy for your own scripts or applications to send requests to it.

Start the server only on 127.0.0.1.

Linux / macOS:

llama-server -hf ggml-org/gemma-3-1b-it-GGUF --host 127.0.0.1 --port 8080

Windows:

llama-server.exe -hf ggml-org/gemma-3-1b-it-GGUF --host 127.0.0.1 --port 8080

Then open a second terminal window and check whether the server responds:

curl http://127.0.0.1:8080/health

Then test a chat request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "user",
        "content": "Write a short summary of what llama.cpp does."
      }
    ]
  }'

In this local example, the Authorization header is a placeholder. Use a real security layer before making the server available to other systems.

Using the API server via an SSH tunnel

Because the server only listens on 127.0.0.1, port 8080 is not directly accessible from the internet. Create an SSH tunnel from your own computer to use the local API on a remote Linux server securely:

ssh -N -L 8080:127.0.0.1:8080 gebruiker@123.123.123.123

Replace gebruiker with your Linux user and 123.123.123.123 with your server’s IP address. As long as the tunnel is active, you can reach the API from your own computer via http://127.0.0.1:8080.

Starting llama-server automatically on Linux

Do you want llama-server to start automatically on a Linux server after a reboot? Create a systemd service. The service below uses the CPU/OpenBLAS build. If you use the CUDA build, replace build/bin with build-cuda/bin and add -ngl 99 to the end of the ExecStart= line.

Step 1

Create the service file:

cd ~/llama.cpp
LLAMA_DIR="$(pwd)"
LLAMA_USER="$(whoami)"

sudo tee /etc/systemd/system/llama-server.service > /dev/null <<EOF
[Unit]
Description=llama.cpp API server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=$LLAMA_DIR
ExecStart=$LLAMA_DIR/build/bin/llama-server -hf ggml-org/gemma-3-1b-it-GGUF --host 127.0.0.1 --port 8080
Restart=on-failure
RestartSec=5
User=$LLAMA_USER

[Install]
WantedBy=multi-user.target
EOF

Step 2

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server

Step 3

Check the status and logs:

systemctl status llama-server
journalctl -u llama-server -f

Stop the log view with ctrl + c.

Updating llama.cpp

Update llama.cpp by fetching the latest code and building it again.

Linux:

cd ~/llama.cpp
git pull
cmake --build build --config Release -j "$(nproc)"

If you use the CUDA build on Linux, replace build with build-cuda. If you use the systemd service from this guide on Linux, restart the service afterwards:

sudo systemctl restart llama-server

macOS:

cd ~/llama.cpp
git pull
cmake --build build --config Release -j "$(sysctl -n hw.logicalcpu)"

Windows:

cd $env:USERPROFILE\llama.cpp
git pull
cmake --build build --config Release

If you use build-cuda, replace build with build-cuda.

Common issues

• cmake: command not found: install the build dependencies again.
• llama.cpp built without libcurl: install libcurl4-openssl-dev or libcurl-devel and build llama.cpp again.
• OpenBLAS is not found: install libopenblas-dev or openblas-devel. If that does not work, use the fallback CPU build without OpenBLAS.
• nvidia-smi or nvcc is not found: first install or repair the NVIDIA driver and CUDA toolkit. Only use the CUDA build afterwards.
• The build fails due to insufficient memory: close other processes, increase your swap memory or use a VPS with more RAM.
• The model responds slowly: use a smaller or more strongly quantized model. CPU inference is noticeably slower than GPU inference.
• Port 8080 is not reachable from your own computer: this is expected when llama-server only listens on 127.0.0.1. Use the SSH tunnel from this guide when the server runs on a remote Linux server.
• A Hugging Face download does not work: check the model name. Some models require you to accept licence terms first or log in with a Hugging Face token.
• HTTPS is not supported: rebuild llama.cpp with -DLLAMA_BUILD_BORINGSSL=ON (see the installation instructions in this article). Without HTTPS support, the -hf option does not work, because llama.cpp cannot download the model directly from Hugging Face.

You have now installed llama.cpp on Linux, macOS or Windows, started a GGUF model and set up a local API server. Use small quantized models on a VPS without a GPU and keep the API server shielded when experimenting with it or connecting your own applications.