Hugging Face hosts community model weights that may not appear in Ollama's library. Ollama can download a compatible public GGUF model by its hf.co repository reference and answer prompts locally.

The repository must contain GGUF weights for an architecture supported by Ollama. A quantization tag selects an existing variant of those weights; it does not convert a repository containing only Safetensors files, and no Modelfile is needed for the direct repository method.

Public, ungated repositories require no Hugging Face sign-in. The first download needs internet access, while inference uses the memory and processors of the machine hosting the Ollama service.

Steps to run a Hugging Face model in Ollama:

  1. List the models already stored in Ollama.
    $ ollama list
    NAME    ID    SIZE    MODIFIED

    Ollama must be installed and its local server running before downloading a model. An empty list has only the header; existing models can remain in place.
    Related: How to install Ollama on Linux
    Related: How to check Ollama server status

  2. Run the public SmolLM2 GGUF model with the Q4_K_M quantization. Example output:
    $ ollama run hf.co/bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M "What is the capital of France? Answer in one sentence."
    ##### snipped #####
    writing manifest
    success
    The capital of France is Paris.

    The SmolLM2-135M-Instruct repository provides this Apache 2.0 licensed model. Its Q4_K_M file is about 105 MB; loading it needs additional memory for the runtime and context. This small model can run on a CPU, but its answers can be inaccurate and the wording varies. For another model, replace bartowski/SmolLM2-135M-Instruct-GGUF with your public GGUF repository and Q4_K_M with a quantization listed in its files. Ollama downloads missing weights before generating the reply.

  3. Confirm that Ollama retains the model under its Hugging Face reference. Example output:
    $ ollama list
    NAME                                                 ID              SIZE      MODIFIED
    hf.co/bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M    f7e5429e6a76    105 MB    Less than a second ago

    The hf.co repository and quantization tag should match the model that answered the prompt. A missing entry or a failed generation does not confirm a usable download.
    Related: How to list Ollama models

Share this guide