A local GGUF file is the direct input that llama.cpp loads for terminal inference. llama-cli provides the terminal path for checking that the file opens, the model evaluates a prompt, and text comes back before the same model is used by a server or application.

Chat models can keep the terminal open for another message after answering. A single-turn run returns control to the shell after one response, which makes it suitable for a non-interactive test.

The starting requirements are an installed llama-cli command and a readable local GGUF file. A small instruction-tuned model limits the memory needed for an initial test; larger models need enough memory for both their weights and working context.

Steps to run a local GGUF model with llama-cli:

  1. Check the local GGUF file at the intended path.
    $ ls -lh models/chat-model.gguf
    -r--r--r-- 1 root root 469M Oct 27 23:48 models/chat-model.gguf

    Here, models/chat-model.gguf is an existing Qwen2.5-0.5B-Instruct Q4_K_M file. If the model is stored elsewhere, substitute its path in both commands. A missing file or unreadable permissions must be resolved before inference.

  2. Run a single-turn prompt against the local model file.
    $ llama-cli -m models/chat-model.gguf \
      -p "What is two plus two?" \
      -n 32 \
      --no-display-prompt \
      --single-turn
    ##### snipped #####
    build      : b11913-7aca0d2ea
    model      : models/chat-model.gguf
    ftype      : Q4_K - Medium
    modalities : text
    ##### snipped #####
    > What is two plus two?
    Two plus two is four.
    
    [ Prompt: 55.3 t/s | Generation: 12.3 t/s ]
    
    Exiting...

    -m selects the GGUF file, -p supplies the prompt, -n limits generated tokens, and --single-turn makes the process return to the shell after the first response. The response and timing values vary with the model and machine; the token budget can also truncate a longer answer.

  3. Check the exit status immediately after llama-cli returns to the shell.
    $ echo $?
    0

    The intended model path, generated text above the timing line, Exiting…, and status 0 together show that the local file loaded and produced a reply. A nonzero status or an empty response means the run needs investigation even if some startup output appeared.

Share this guide