Local GGUF models need an HTTP API before a chat interface can use them. llama.cpp supplies that API through llama-server, allowing Open WebUI to send prompts to an existing model server.

The server exposes model listing and chat completions under /v1. Open WebUI uses its OpenAI API connection layer for those requests, with a llama.cpp provider option under Advanced. The model's API ID determines which entry appears in the chat model picker.

An Open WebUI administrator account and an already-serving llama.cpp endpoint are required. The endpoint must be reachable from the Open WebUI backend, which may run on a different machine or inside a container. A browser reaching the model server does not establish that the backend can reach it.

Steps to connect llama.cpp to Open WebUI:

  1. Open your profile menu in the lower-left corner of Open WebUI.
  2. Select Settings.
  3. Select Connections under Admin → AI.
  4. Click the plus button beside Manage OpenAI API Connections.

    The dialog defaults to External for Connection Type, Bearer for Auth, and Chat Completions for API Type.

  5. Enter your llama.cpp endpoint ending in /v1 in URL.

    https://llm.example.com/v1 represents an existing endpoint reachable from the backend. For a container, localhost refers to that container rather than the Docker host.
    Related: How to fix Open WebUI localhost connection errors in Docker

    A model endpoint can accept prompts from anyone who can reach it unless access is restricted. A private network or authenticated HTTPS endpoint protects model access; disabling certificate verification exposes prompts and credentials to interception.

  6. Enter the endpoint's API key in the masked key field.

    For an intentionally keyless server on a trusted private network, Auth must be None instead; a made-up API key is unnecessary. A key configured on an authenticating proxy belongs here when that proxy supplies the URL.

  7. Expand Advanced.
  8. Select llama.cpp in Provider.

    Model IDs can remain empty when the server advertises its model through /v1/models. The returned ID may come from the server's --alias setting rather than the GGUF filename.

  9. Click Verify Connection beside URL.

    The circular-arrow button tests the endpoint from the backend. A connection error requires checking the URL, authentication, certificate trust, and network route before saving.

  10. Connect llama.cpp by clicking Save in Add Connection.
  11. Click Back to return to the chat.
  12. Open New Chat.
  13. Open the model picker in the message box.
  14. Select the model ID advertised by your llama.cpp server.

    The server used here advertises company-chat. A model missing from this list indicates a discovery or access problem.
    Related: How to refresh the Open WebUI model list

  15. Enter What is 2 + 2? Answer in one short sentence. in the message box.
  16. Send the message with the upward-arrow button.
  17. Confirm that the selected model returns an answer in the chat.

    A reply under company-chat exercises the saved connection and the server's chat completion route. A saved URL or successful model listing alone does not show that inference works. An “exceeds the available context size” error means the server’s context limit is too small for the request.

Share this guide