GGUF repositories on Hugging Face publish model weights ready for local inference. llama.cpp can download and run a model by repository name, keeping the weights in a local cache for later use without an internet connection.

The -hf option points llama-cli at a Hugging Face repository, and a suffix such as :Q4_K_M selects a matching quantization. The -hff option selects an exact filename instead, avoiding ambiguity when a repository publishes several GGUF files.

Public, ungated models need no account or access token. The download occupies disk space in addition to the memory needed for inference, so a small quantized model is a suitable first CPU test.

Steps to run a Hugging Face GGUF model with llama.cpp:

  1. Check the Hugging Face model cache before downloading.
    $ llama-cli --cache-list
    number of models in cache: 0

    An empty cache reports zero models; an existing cache may already contain entries. This requires a recent llama-cli installation on PATH.
    Related: How to install llama.cpp with Homebrew
    Related: How to build llama.cpp from source with CMake

  2. Download the public Qwen2.5-0.5B-Instruct Q4_K_M file with a single-turn prompt.
    $ llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF \
      -hff qwen2.5-0.5b-instruct-q4_k_m.gguf \
      -p "What is two plus two?" -n 32 --single-turn
    ##### snipped #####
    build      : b11913-7aca0d2ea
    model      : Qwen/Qwen2.5-0.5B-Instruct-GGUF
    ftype      : Q4_K - Medium
    modalities : text
    ##### snipped #####
    > What is two plus two?
    Two plus two equals four.
    ##### snipped #####
    Exiting...

    The publisher's repository lists the exact filename. This file is about 491 MB; larger models need more disk space and memory. The first run needs internet access. -hff overrides any quantization suffix on -hf. -n 32 limits generated tokens, and --single-turn exits after the response. Wording and timing vary between runs.

  3. List the downloaded model in the cache.
    $ llama-cli --cache-list
    number of models in cache: 1
       1. Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M

    Current builds use the standard Hugging Face cache, normally ~/.cache/huggingface/hub on Linux and macOS. HF_HUB_CACHE or HF_HOME can redirect it; LLAMA_CACHE takes precedence when set.

  4. Run the downloaded model in offline mode.
    $ llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF \
      -hff qwen2.5-0.5b-instruct-q4_k_m.gguf --offline \
      -p "What is two plus two?" -n 32 --single-turn
    ##### snipped #####
    > What is two plus two?
    Two plus two equals four.
    ##### snipped #####
    Exiting...

    A generated answer followed by Exiting… shows that the cached model remains usable. --offline prevents network access and fails when the required file is absent. If a cache miss occurs, repeat the online download with the same repository, filename, and cache settings.

Share this guide