A model's next-token choices depend on the sampling controls as well as the prompt. In llama.cpp, generation parameters on a completion request let a client adjust randomness, repetition penalties, and output length for that call.

The native /completion endpoint applies sampling overrides to the current generation task. A client can tune its own request without changing the server startup command or the defaults used by another client.

A fixed seed helps isolate a parameter change, but identical text is not guaranteed across different models, builds, or backends. The returned generation_settings shows the effective values; the generated content and token count show that the request also produced text.

Steps to set llama.cpp generation parameters:

  1. Start llama-server with a local text-generation GGUF model.
    $ llama-server -m models/chat-model.gguf --alias local-chat --ctx-size 2048 --host 127.0.0.1 --port 8080

    models/chat-model.gguf must point to an existing model file. An instruction model such as Qwen2.5-0.5B-Instruct can answer the robot-painter prompt; generated wording depends on the model. The request and response checks require curl and Python 3.
    Related: How to start the llama.cpp server

  2. Check that the local server has finished loading from another terminal.
    $ curl -fsS http://127.0.0.1:8080/health
    {"status":"ok"}

    The -f option makes an HTTP error fail the curl command. The loopback listener accepts local connections; exposing it to other computers requires separate access controls.

  3. Send a completion request with a temperature of 0.35.
    $ curl -fsS http://127.0.0.1:8080/completion \
      -H 'Content-Type: application/json' \
      -d '{
      "prompt": "Write one short sentence about a robot painter.",
      "n_predict": 40,
      "temperature": 0.35,
      "top_p": 0.80,
      "repeat_penalty": 1.08,
      "seed": 42,
      "stop": [
        "\n"
      ]
    }' \
      -o generation-cool.json

    The response is saved in generation-cool.json. temperature controls randomness, top_p limits the candidate pool by cumulative probability, and repeat_penalty penalizes repeated tokens. n_predict caps generated tokens rather than words, while stop ends generation at a newline if one appears. Other sampler settings retain their server defaults.

  4. Send the same prompt with only temperature raised to 0.80.
    $ curl -fsS http://127.0.0.1:8080/completion \
      -H 'Content-Type: application/json' \
      -d '{
      "prompt": "Write one short sentence about a robot painter.",
      "n_predict": 40,
      "temperature": 0.80,
      "top_p": 0.80,
      "repeat_penalty": 1.08,
      "seed": 42,
      "stop": [
        "\n"
      ]
    }' \
      -o generation-warm.json

    The prompt, seed, token budget, stop string, and other sampling values remain unchanged, so the comparison isolates the temperature change.

  5. Verify both responses contain text and the requested generation parameters.
    $ python3 - <<'PYTHON'
    import json
    import math
    for name, temperature in [("cool", 0.35), ("warm", 0.80)]:
        with open("generation-" + name + ".json") as response_file:
            response = json.load(response_file)
        settings = response["generation_settings"]
        expected = {"temperature": temperature, "top_p": 0.80,
                    "repeat_penalty": 1.08, "seed": 42, "n_predict": 40}
        assert all(math.isclose(settings[k], v, rel_tol=1e-6) for k, v in expected.items())
        assert settings["stop"] == ["\n"]
        assert response["content"].strip() and response["tokens_predicted"] > 0
        print(name, "temperature:", settings["temperature"])
        print("top_p:", settings["top_p"], "repeat_penalty:", settings["repeat_penalty"])
        print("seed:", settings["seed"], "n_predict:", settings["n_predict"])
        print("content:", response["content"])
        print("tokens_predicted:", response["tokens_predicted"], "stop_type:", response["stop_type"])
    PYTHON
    cool temperature: 0.3499999940395355
    top_p: 0.800000011920929 repeat_penalty: 1.0800000429153442
    seed: 42 n_predict: 40
    content:  A robot painter can paint on walls and other surfaces using a variety of tools and techniques, including brushes, rollers, and spray paint.
    tokens_predicted: 28 stop_type: eos
    warm temperature: 0.800000011920929
    top_p: 0.800000011920929 repeat_penalty: 1.0800000429153442
    seed: 42 n_predict: 40
    content:  A robot is able to paint on walls, using its brush to create vivid colors and textures. 
    tokens_predicted: 20 stop_type: word

    The assertions fail if a setting differs, a response is empty, or no tokens were generated. Floating-point values such as 0.3499999940395355 represent the requested 0.35 within the comparison tolerance. A stop_type of eos means the model ended the response, word means a stop string was reached, and limit means the token budget ended generation. Different temperatures can still produce the same wording.
    Related: How to call the llama.cpp chat completions API

Share this guide