How to call the llama.cpp rerank API

Reranking in llama.cpp scores candidate passages against a search query after an initial retrieval step. It helps a local search or RAG pipeline put the most relevant text first before sending context to a chat or completion model.

The llama-server rerank endpoint is separate from chat completions and embeddings. Start the server with a reranker GGUF model, enable reranking, and use rank pooling so requests return document scores instead of generated text.

A public BGE reranker GGUF works for a local smoke test with the alias local-reranker. Keep the listener on localhost for local development, and replace the model source with an approved reranker when query text or documents include private data.

1. Start llama-server with a reranker model and a short API alias.

   $ llama-server \
     -hf TheOSExplorer/bge-reranker-v2-m3-Q2_K-GGUF:Q2_K \
     --alias local-reranker \
     --embedding \
     --pooling rank \
     --reranking
   ##### snipped #####
   srv  llama_server: model loaded
   srv  llama_server: listening on http://127.0.0.1:8080

   --reranking enables the rerank endpoint. The current llama.cpp server documentation also requires --embedding and --pooling rank with reranker models, so keep the three flags together for this endpoint.
   Related: How to start the llama.cpp server

2. Check the server health endpoint from another terminal.

   $ curl http://127.0.0.1:8080/health
   {"status":"ok"}

   A 503 response with Loading model means the model is still loading. Wait for {"status":"ok"} before sending rerank requests.

3. List the model ID clients should send in the request body.

   $ curl http://127.0.0.1:8080/v1/models
   {
     "object": "list",
     "data": [
       {
         "id": "local-reranker",
         "object": "model",
         "owned_by": "llamacpp",
         "meta": {
           "n_ctx": 8192
         }
       }
     ]
   }

4. Send a rerank request with the query and candidate documents.

   $ curl http://127.0.0.1:8080/v1/rerank \
     --header "Content-Type: application/json" \
     --data '{
       "model": "local-reranker",
       "query": "Which document explains llama.cpp reranking?",
       "top_n": 3,
       "documents": [
         "llama.cpp can run a local reranker model through llama-server.",
         "A weather station reports wind speed and rainfall.",
         "Reranking scores candidate passages after an initial search step."
       ]
     }'
   {
     "model": "local-reranker",
     "object": "list",
     "usage": {
       "prompt_tokens": 90,
       "total_tokens": 90
     },
     "results": [
       {
         "index": 0,
         "relevance_score": -0.23934173583984375
       },
       {
         "index": 2,
         "relevance_score": -2.8770065307617188
       },
       {
         "index": 1,
         "relevance_score": -10.827690124511719
       }
     ]
   }

   documents keeps the original candidate order. Each returned index points back to that zero-based array position.

5. Use the returned indexes to reorder the original candidate list.

   0  llama.cpp can run a local reranker model through llama-server.
   2  Reranking scores candidate passages after an initial search step.
   1  A weather station reports wind speed and rainfall.

   The score scale depends on the reranker model. Treat higher scores as more relevant within the same request instead of comparing absolute values across different models.