Keeping several language models resident can exhaust memory before any client sends a prompt. The llama.cpp server router can load a known GGUF model on demand while keeping its HTTP listener running.

Router mode differs from a single-model llama-server launch. It runs without -m, takes its model catalog from a local directory, and starts a child server for each loaded model. The catalog ID selects the destination for inference requests.

A successful /models/load response means the load request was accepted, not that loading has finished. The model's status moves from loading to loaded independently of router health. Each loaded child needs enough RAM or GPU memory for its model and context; a model-count limit does not cap memory use.

Steps to load a model with the llama.cpp server router:

  1. Start the router with the local chat-model directory on loopback port 8080.
    $ llama-server --models-dir ./models --no-models-autoload --models-max 1 --host 127.0.0.1 --port 8080

    This launch assumes llama-server is on PATH and ./models contains an existing chat-capable GGUF, such as chat-model.gguf. The directory supplies only the chat models intended for this router. The router runs without -m; --no-models-autoload prevents inference requests from loading an unloaded model automatically, and --models-max 1 limits the router to one loaded child. This terminal stays occupied while the router runs. Loopback limits access to the same machine but adds no authentication.
    Related: How to start the llama.cpp server

  2. Request router health from a second terminal.
    $ curl --silent --show-error --fail-with-body http://127.0.0.1:8080/health
    {"status":"ok"}

    A ready router can still have every model unloaded. /health describes the front-facing service, not a specific model's readiness. The requests use curl 7.76.0 or newer for --fail-with-body, which preserves error JSON while returning a nonzero exit status on HTTP errors.
    Related: How to check llama.cpp server health

  3. Discover the unloaded model ID in the refreshed router catalog.
    $ curl --silent --show-error --fail-with-body 'http://127.0.0.1:8080/models?reload=1'
    {"data":[{"id":"chat-model","aliases":[],"tags":[],"object":"model","owned_by":"llamacpp","created":1793149770,"status":{"value":"unloaded",
    ##### snipped #####

    The id in data is chat-model here, from chat-model.gguf. Its status.value is unloaded. A different filename produces a different catalog ID. Quoting the query URL avoids wildcard expansion in zsh. The shortened output omits arguments and metadata. A catalog refresh can unload a running model if its source has changed or disappeared.

  4. Load the discovered model through the router API.
    $ curl --silent --show-error --fail-with-body http://127.0.0.1:8080/models/load \
      --header 'Content-Type: application/json' \
      --data '{"model":"chat-model"}'
    {"success":true}

    If the catalog returned a different id, the model value must match that ID in both this request and the later chat request. success acknowledges the asynchronous load request.

  5. Check the model status until it reaches loaded.
    $ curl --silent --show-error --fail-with-body http://127.0.0.1:8080/models
    {"data":[{"id":"chat-model","aliases":[],"tags":[],"object":"model","owned_by":"llamacpp","created":1793149772,"status":{"value":"loaded",
    ##### snipped #####

    If status.value is still loading, wait a few seconds and repeat this request. A failed child can return to unloaded with status.failed set to true and an exit_code; the router terminal logs identify the file, argument, or memory error. This status check does not need another catalog refresh.

  6. Send a chat request routed to the loaded model ID.
    $ curl --silent --show-error --fail-with-body http://127.0.0.1:8080/v1/chat/completions \
      --header 'Content-Type: application/json' \
      --data '{"model":"chat-model","messages":[{"role":"user","content":"What is two plus two?"}],"max_tokens":64}'
    {"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Two plus two is four."}}],"created":1793149774,"model":"chat-model","system_fingerprint":"b11913-7aca0d2ea","object":"chat.completion","usage":{"completion_tokens":7,"prompt_tokens":35,"total_tokens":42,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-U4d9klEdRguH7rqhbj2VH1pH726Bxr8I","timings":{"cache_n":0,"prompt_n":35,"prompt_ms":593.02,"prompt_per_token_ms":16.94342857142857,"prompt_per_second":59.019931874135786,"predicted_n":7,"predicted_ms":591.193,"predicted_per_token_ms":98.53216666666667,"predicted_per_second":10.148969964123392}}

    The response identifies chat-model and contains an assistant answer under choices. The wording can vary. A nonempty answer and finish_reason set to stop show that the loaded child handled the routed request.
    Related: How to call the llama.cpp chat completions API

Share this guide