Docker keeps the llama.cpp executable and its libraries separate from the host's software. The official CPU server image serves a local GGUF model over HTTP without building the C++ project on the host.

The server image runs llama-server, while Docker maps its container port to a port on the host. A read-only bind mount keeps model files outside the image and prevents the container from changing them.

Docker must already be running, and a chat-tuned GGUF model must be available locally. The CPU image needs enough host RAM for the model and its context; a GPU image requires a separately configured GPU container runtime.

Steps to run llama.cpp with Docker:

  1. Check that the local chat model exists at models/chat-model.gguf.
    $ ls -lh models/chat-model.gguf

    The shell syntax assumes Bash or a compatible shell. The model used here is Qwen2.5-0.5B-Instruct Q4_K_M, saved as models/chat-model.gguf. A different chat-tuned GGUF needs its own filename and a compatible chat template.

  2. Start the CPU server container with the model directory mounted read-only.

    The published port accepts requests without an API key. Binding the host port to 127.0.0.1 limits access to the Docker host; it does not add authentication. Publishing on all host interfaces can expose prompts, responses, and inference capacity to other machines.

    $ docker run --detach --name llama-cpp-server \
      --publish 127.0.0.1:8080:8080 \
      --mount "type=bind,source=$PWD/models,target=/models,readonly" \
      ghcr.io/ggml-org/llama.cpp:server \
      --model /models/chat-model.gguf \
      --alias local-chat \
      --ctx-size 2048 \
      --host 0.0.0.0 \
      --port 8080

    Docker prints a container ID and leaves the server running in the background. The name llama-cpp-server and host port 8080 must be unused. --host 0.0.0.0 applies inside the container so Docker can forward traffic to it. The 2048-token context limits the combined prompt and response budget. The server tag tracks upstream updates; an image digest can pin a specific build.

  3. Request readiness through the published host port.
    $ curl --fail --silent --show-error http://127.0.0.1:8080/health
    {"status":"ok"}

    For this single-model launch, ok means the model is ready. During loading, the endpoint can return HTTP 503 with Loading model; a later request should return ok. If the container exits or readiness keeps failing, docker logs llama-cpp-server shows startup errors.
    Related: How to check llama.cpp server health

  4. Confirm that the container serves the model ID local-chat.
    $ curl --fail --silent --show-error http://127.0.0.1:8080/v1/models

    The id under data should match the local-chat alias supplied at startup. Other metadata fields vary by build and model.

  5. Send a chat request through the Docker host's published port.
    $ curl --fail --silent --show-error http://127.0.0.1:8080/v1/chat/completions \
      --header 'Content-Type: application/json' \
      --data '{"model":"local-chat","messages":[{"role":"user","content":"What is two plus two?"}],"max_tokens":64,"temperature":0}'
    {"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Two plus two is four."}}],"created":1793144985,"model":"local-chat","system_fingerprint":"b11913-7aca0d2ea","object":"chat.completion","usage":{"completion_tokens":7,"prompt_tokens":35,"total_tokens":42,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-uQ8lLqFJuKgmf6wmgbV9U5hYWbJSdICp","timings":{"cache_n":0,"prompt_n":35,"prompt_ms":1098.268,"prompt_per_token_ms":31.379085714285715,"prompt_per_second":31.868359999562948,"predicted_n":7,"predicted_ms":1539.076,"predicted_per_token_ms":256.5126666666667,"predicted_per_second":3.8984429618810243}}

    A nonempty assistant message with object set to chat.completion shows that the container can generate through the published port. Wording and timings can vary. The container remains available for further requests.
    Related: How to call the llama.cpp chat completions API

Share this guide