Local AI setup: Ollama, LM Studio, llama.cpp and vLLM

The tools

• Ollama — command-line tool and local server: ollama run downloads a model and starts a chat. The easiest start. See our Ollama guide.
• LM Studio — a desktop app with a graphical interface for finding, downloading and chatting with models.
• llama.cpp — the open C/C++ engine (2023) behind many local tools; runs models efficiently on ordinary CPUs and GPUs using the GGUF file format.
• vLLM — a serving library from UC Berkeley researchers, built for many users at once on server GPUs.

Quantisation: fitting models in memory

Model weights are normally stored as 16-bit numbers. Quantised versions store them in 8, 5 or 4 bits, cutting memory use by a half or more with a small loss of quality. As a rough guide, a 7- to 8-billion-parameter model at 4 bits needs around 5 GB of memory; the same model at 16 bits needs around 16 GB.

Which one?

• Trying models on a laptop: LM Studio or Ollama.
• Connecting local models to other programs: Ollama's local API.
• Serving a team or an application: vLLM on a GPU server.

At ZIBADIS

ZIBADIS AI Lab runs 56 open models in 17 categories on its own servers — text, code, vision, speech, OCR, translation, 3D and more — with no external API.

Tools for this

• ZIBADIS AI Lab