How to set an OpenAI-compatible base URL in LlamaIndex

Many local and hosted inference servers expose the same /v1 shape as the OpenAI API, but LlamaIndex still needs to know which endpoint should receive its LLM calls. Setting an OpenAI-compatible base URL routes the OpenAI-style LLM wrapper to a local server, gateway, or third-party provider without changing the rest of the query or agent code.

The OpenAILike integration is the closest fit for non-OpenAI servers that implement OpenAI chat or completion routes. Its api_base value should include the provider's API root, usually ending in /v1, while model must match a model name exposed by that endpoint.

Use a placeholder api_key only when the endpoint does not require authentication; keep real provider tokens in environment variables or a secret store. Check the model list first, then make one LlamaIndex completion call so endpoint routing and model selection fail before larger RAG code is involved.

1. Identify the endpoint root and model name exposed by the compatible server.

   Endpoint: http://127.0.0.1:8000/v1
   Model: local-chat

   Replace the endpoint and model with values from LM Studio, llama.cpp, vLLM, an API gateway, or another compatible provider. Keep the API root, not a full chat-completions path.

2. Check that the endpoint exposes the expected model.

   $ curl http://127.0.0.1:8000/v1/models
   {"object": "list", "data": [{"id": "local-chat", "object": "model"}]}

   If the server requires authentication, use the same bearer token or provider header that the Python client will use. Do not paste real tokens into saved transcripts, screenshots, or shared shell history.

3. Install the OpenAI-compatible LlamaIndex LLM integration in the active Python environment.

   $ python3 -m pip install llama-index-llms-openai-like

4. Create a short LlamaIndex smoke-test script.

   $ $EDITOR check_llamaindex_base_url.py

   from llama_index.llms.openai_like import OpenAILike
   
   llm = OpenAILike(
       model="local-chat",
       api_base="http://127.0.0.1:8000/v1",
       api_key="not-needed",
       is_chat_model=True,
       is_function_calling_model=False,
       temperature=0,
   )
   
   response = llm.complete("Reply with one short confirmation sentence.")
   
   print("Endpoint: http://127.0.0.1:8000/v1")
   print("Model: local-chat")
   print("Reply:", str(response))

   Use is_chat_model=True for endpoints that implement POST /v1/chat/completions. Set is_function_calling_model=True only when the selected server and model support OpenAI-compatible tool calls.

5. Run the script to confirm LlamaIndex reaches the configured base URL.

   $ python3 check_llamaindex_base_url.py
   Endpoint: http://127.0.0.1:8000/v1
   Model: local-chat
   Reply: LlamaIndex reached the custom endpoint.

   A 404 or route error usually means api_base included the wrong path, missed /v1, or pointed at a server that supports a different OpenAI-compatible route set.

6. Remove the temporary smoke-test script after copying the working settings into the application code.

   $ rm check_llamaindex_base_url.py