Reranking in llama.cpp scores candidate passages against a search query after an initial retrieval step. The rerank API returns their relevance order so a local search or RAG pipeline can put the closest matches into a model's context.

The llama-server rerank endpoint is separate from chat completions and embeddings. A reranker evaluates each query–passage pair rather than generating an answer or returning a reusable document vector.

Jina Reranker v1 turbo English is suitable for English query–passage comparisons. It needs its own reranker GGUF file; a chat or embedding model cannot substitute for it.

Steps to call the llama.cpp rerank API:

The starting requirements are llama-server on PATH, curl with --fail-with-body support, and a local copy of ggml-org's Jina reranker GGUF at models/reranker-model.gguf. The upstream filename is Jina-Bert-Implementation-38M-F16.gguf, converted from Jina AI's reranker.

The server has no API-key checking enabled. A 127.0.0.1 listener limits network access to this machine, but other local processes can still submit queries and documents. Remote access needs an appropriate access policy before the listener is exposed.

  1. Start llama-server with the Jina reranker on 127.0.0.1:8080.
    $ llama-server -m models/reranker-model.gguf --alias local-reranker --reranking --host 127.0.0.1 --port 8080

    --reranking enables the endpoint and selects rank pooling. The --embedding and --pooling rank switches are unnecessary with this option. The server stays in the foreground; the HTTP requests run in another terminal.
    Related: How to start the llama.cpp server

  2. Check that the reranker has finished loading.
    $ curl --fail-with-body http://127.0.0.1:8080/health
    {"status":"ok"}

    A loading model can return HTTP 503. The --fail-with-body option retains the error body and makes HTTP errors produce a nonzero curl exit status.
    Related: How to check llama.cpp server health

  3. Save the query and three candidate passages in rerank-request.json.
    $ cat > rerank-request.json <<'JSON'
    {
      "model": "local-reranker",
      "query": "What is the capital of France?",
      "top_n": 2,
      "documents": [
        "Paris is the capital of France.",
        "Bananas grow in warm climates.",
        "Berlin is the capital of Germany."
      ]
    }
    JSON

    documents preserves the original candidate order. Its zero-based positions are 0 for Paris, 1 for bananas, and 2 for Berlin. The top_n value requests only the two highest-ranked results.

  4. Submit the passages to /v1/rerank for their relevance order.
    $ curl --fail-with-body http://127.0.0.1:8080/v1/rerank \
      --header "Content-Type: application/json" \
      --data-binary @rerank-request.json
    {"model":"local-reranker","object":"list","usage":{"prompt_tokens":56,"total_tokens":56},"results":[{"index":0,"relevance_score":-0.03818938508629799},{"index":2,"relevance_score":-0.07548195123672485}]}

    The two results entries point to original indexes 0 and 2, placing Paris before Berlin and excluding the banana passage. They arrive in descending relevance_score order, which can differ from the input order. Higher scores mean a closer match within the same request; scores can be negative and have no universal relevance cutoff across models.

Share this guide