Llama.cpp is a popular open-source software framework that allows you to run large language models (LLMs, such as Llama by Meta) locally on your own computer. It is designed in C/C++ and excels in CPU efficiency, meaning you do not need an expensive graphics card (GPU) to use AI.
What is Llama.cpp used for?
Llama.cpp is used for two things: A) hosting LLMs locally and B) it is used by many popular AI apps such as Ollama and LM Studio as an inference backend. However, applications like Ollama also add performance overhead. For this reason, in terms of hardware usage, it is more efficient to host an LLM yourself via Llama.cpp instead of using an application like Ollama.
Which LLMs are available for Llama.cpp?
All self-hosted models available via Hugging Face can be used via Llama.cpp. This not only makes many more models available than via Ollama, for example, but also makes new LLMs available faster via Llama.cpp.
Can I use LLama.cpp myself?
Certainly! Some technical knowledge is required for using the command line and SSH, but in this guide, we explain exactly how to install Llama.cpp and connect it to popular AI applications such as Codex, Claude Code, and OpenWebUI.
Which LLM is best to use with Llama.cpp?
This depends on your use case and the resources available on your server. In this article, we will delve deeper into this and you will find a simple cheatsheet.