How to run llama.cpp with Docker

Docker is a good fit for llama.cpp when the host should provide only the container runtime and a GGUF model file. The official images include separate targets for the command-line tools and llama-server, so a local model can be served without building the C++ project on the host.

The server image runs llama-server, and publishing container port 8080 lets local clients call the HTTP API through localhost. A read-only bind mount keeps the model file outside the image while making it available inside the container as /models/model.gguf.

The sample model is a small public GGUF file used to prove the container path. Replace it with a model that fits the host hardware before exposing the server to other users, and use a GPU-tagged image only after the host GPU container runtime is installed.

1. Open a terminal in the working directory that will hold the model files.
2. Create a local model directory.

   $ mkdir -p models

3. Download a small GGUF model for the first container run.

   $ curl --fail --location \
     --output models/tinygemma3-Q8_0.gguf \
     https://huggingface.co/ggml-org/tinygemma3-GGUF/resolve/main/tinygemma3-Q8_0.gguf

   Use an existing GGUF file instead when one is already available. Copy it into models and use that filename in the Docker command.

4. Start the llama-server container with the model directory mounted read-only.

   $ docker run --detach --name llama-cpp-server \
     --publish 127.0.0.1:8080:8080 \
     --volume "$PWD/models:/models:ro" \
     ghcr.io/ggml-org/llama.cpp:server \
     --model /models/tinygemma3-Q8_0.gguf \
     --host 0.0.0.0 \
     --port 8080
   709ce8afa7be5581d4a5d964a9da6705ab4719686aea86ace03811283b0ed8c8

   --publish 127.0.0.1:8080:8080 keeps the server reachable from the local host only. Use --publish 8080:8080 only when the host firewall and network policy should allow remote clients.

5. Check the server readiness endpoint.

   $ curl http://localhost:8080/health
   {"status":"ok"}

   If the model is still loading, the health endpoint can return a 503 response with Loading model. Wait a few seconds and run the same check again.
   Related: How to check llama.cpp server health

6. Confirm that the server reports the mounted model.

   $ curl http://localhost:8080/v1/models
   {"models":[{"name":"/models/tinygemma3-Q8_0.gguf","model":"/models/tinygemma3-Q8_0.gguf","capabilities":["completion"]}],"object":"list","data":[{"id":"/models/tinygemma3-Q8_0.gguf","object":"model","owned_by":"llamacpp"}]}

   The exact metadata fields vary by llama.cpp build and model. Look for the mounted model path and the model-list response object.
   Related: How to call the llama.cpp chat completions API

7. Remove the test container when the Docker run is no longer needed.

   $ docker rm --force llama-cpp-server
   llama-cpp-server