How to start the llama.cpp server

The llama.cpp server turns a loaded GGUF model into a local HTTP service for browser tests, scripts, and OpenAI-compatible clients. Local clients can reuse that process instead of loading the model for every prompt.

The foreground llama-server process loads one model and serves the health endpoint, model listing, web UI, and OpenAI-compatible API routes from the same listener. The model stays available while that process runs; closing its terminal stops the service.

A loopback listener limits connections to the same machine, but local access has no API-key protection unless authentication is configured. A remote listener can expose prompts, responses, and inference capacity to other clients, so remote access needs authentication, network restrictions, and encrypted transport before exposure.

1. Check the local chat model file before loading it.

   $ ls -lh models/chat-model.gguf
   -r--r--r-- 1 root root 469M Oct 27 23:32 models/chat-model.gguf

   This launch assumes llama-server is installed and available on PATH. The file shown is Qwen2.5-0.5B-Instruct Q4_K_M, already saved as models/chat-model.gguf. A different chat-tuned GGUF needs its own file path and a compatible chat template. Related: How to install llama.cpp with Homebrew
   Related: How to build llama.cpp from source with CMake

2. Start llama-server with the local model and alias on loopback port 8080.

   $ llama-server -m models/chat-model.gguf --alias local-chat --ctx-size 2048 --host 127.0.0.1 --port 8080
   ##### snipped #####
   0.00.704.336 I srv  server_run_i: model loaded
   0.00.704.363 I srv  server_on_li: listening on http://127.0.0.1:8080

   The explicit host and port keep the listener at 127.0.0.1:8080 across builds with different defaults. The alias local-chat becomes the model ID for API clients. The 2048-token context limits the combined prompt and response budget. This terminal stays occupied while the server runs; Ctrl+C stops it when it is no longer needed.

3. Request the health endpoint from a second terminal.

   $ curl --fail --silent --show-error http://127.0.0.1:8080/health
   {"status":"ok"}

   For this single-model launch, status set to ok means the server is ready. During loading, the endpoint can return HTTP 503 with Loading model, and --fail makes curl exit nonzero.

4. Confirm that the model listing contains local-chat.

   $ curl --fail --silent --show-error http://127.0.0.1:8080/v1/models
   {"models":[{"name":"local-chat","model":"local-chat","modified_at":"","size":"","digest":"","type":"model","description":"","tags":[""],"capabilities":["completion"],"parameters":"","details":{"parent_model":"","format":"gguf","family":"","families":[""],"parameter_size":"","quantization_level":""}}],"object":"list","data":[{"id":"local-chat","aliases":["local-chat"],"tags":[],"object":"model","architecture":{"input_modalities":["text"],"output_modalities":["text"]},"created":1793143964,"owned_by":"llamacpp","meta":{"vocab_type":2,"n_vocab":151936,"n_ctx":2048,"n_ctx_train":32768,"n_embd":896,"n_params":630167424,"size":485452288,"ftype":"Q4_K - Medium"}}]}

   The id under data matches the startup alias and the model field in the chat request.

5. Send a short chat request to the running model.

   $ curl --fail --silent --show-error http://127.0.0.1:8080/v1/chat/completions \
     --header "Content-Type: application/json" \
     --data '{"model":"local-chat","messages":[{"role":"user","content":"What is two plus two?"}],"max_tokens":32}'
   {"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Two plus two is four."}}],"created":1793143965,"model":"local-chat","system_fingerprint":"b11913-7aca0d2ea","object":"chat.completion","usage":{"completion_tokens":7,"prompt_tokens":35,"total_tokens":42,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-JqxREc8wrJF9WC9CfLvkTirRafro9Ezh","timings":{"cache_n":0,"prompt_n":35,"prompt_ms":449.371,"prompt_per_token_ms":12.839171428571428,"prompt_per_second":77.88664600074327,"predicted_n":7,"predicted_ms":288.049,"predicted_per_token_ms":48.00816666666666,"predicted_per_second":20.82978937611309}}

   The assistant message under choices contains the generated answer. Its wording can vary; a nonempty response with model set to local-chat and object set to chat.completion shows that the loaded server can process a chat request.