A local GGUF file is the direct input that llama.cpp loads for terminal inference. llama-cli provides the terminal path for checking that the file opens, the model evaluates a prompt, and text comes back before the same model is used by a server or application.
Chat models can keep the terminal open for another message after answering. A single-turn run returns control to the shell after one response, which makes it suitable for a non-interactive test.
The starting requirements are an installed llama-cli command and a readable local GGUF file. A small instruction-tuned model limits the memory needed for an initial test; larger models need enough memory for both their weights and working context.
Steps to run a local GGUF model with llama-cli:
- Check the local GGUF file at the intended path.
$ ls -lh models/chat-model.gguf -r--r--r-- 1 root root 469M Oct 27 23:48 models/chat-model.gguf
Here, models/chat-model.gguf is an existing Qwen2.5-0.5B-Instruct Q4_K_M file. If the model is stored elsewhere, substitute its path in both commands. A missing file or unreadable permissions must be resolved before inference.
- Run a single-turn prompt against the local model file.
$ llama-cli -m models/chat-model.gguf \ -p "What is two plus two?" \ -n 32 \ --no-display-prompt \ --single-turn ##### snipped ##### build : b11913-7aca0d2ea model : models/chat-model.gguf ftype : Q4_K - Medium modalities : text ##### snipped ##### > What is two plus two? Two plus two is four. [ Prompt: 55.3 t/s | Generation: 12.3 t/s ] Exiting...
-m selects the GGUF file, -p supplies the prompt, -n limits generated tokens, and --single-turn makes the process return to the shell after the first response. The response and timing values vary with the model and machine; the token budget can also truncate a longer answer.
- Check the exit status immediately after llama-cli returns to the shell.
$ echo $? 0
The intended model path, generated text above the timing line, Exiting…, and status 0 together show that the local file loaded and produced a reply. A nonzero status or an empty response means the run needs investigation even if some startup output appeared.