A Blog post by ggml-org on Hugging Face
Back to Articles
llama.cpp server now ships with router mode, which lets you dynamically load, unload, and switch between multiple models without restarting.
Reminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs locally.
This feature was a popular request to bring Ollama-style model management to llama.cpp. It uses a multi-process architecture where each model runs in its own process, so if one model crashes, others remain unaffected.
Quick Start
A Blog post by ggml-org on Hugging Face
Llama 2 is the first open source language model of the same caliber as OpenAI’s models. Learn how to run it in the cloud with one…
A fast TUI, CLI, daemon, and OpenAI-compatible proxy for running local LLMs via llama.cpp, in one Rust binary
A roundup of recent developments from the llamaverse.
Llama 3.1 405B: is the most powerful open-source language model from Meta. Learn how to run it in the cloud with one line of code.
Llama 3 is the latest language model from Meta. Learn how to run it in the cloud with one line of code.
