How to set Ollama request parallelism

Ollama request parallelism controls how many requests the server can process concurrently. Tuning it helps busy local APIs balance throughput, latency, and memory use.

The relevant server variables include OLLAMA_NUM_PARALLEL for parallel requests, OLLAMA_MAX_QUEUE for queued requests, and OLLAMA_MAX_LOADED_MODELS for loaded model capacity. Set them before the server starts.

Increase values gradually and verify with real workload behavior. Higher parallelism can load more model state into memory and may reduce reliability on small GPUs.

1. Choose conservative parallelism and queue values.

   $ printf '%s\n' 'OLLAMA_NUM_PARALLEL=2 OLLAMA_MAX_QUEUE=64'
   OLLAMA_NUM_PARALLEL=2 OLLAMA_MAX_QUEUE=64

2. Test the settings with a foreground server.

   $ OLLAMA_NUM_PARALLEL=2 OLLAMA_MAX_QUEUE=64 ollama serve

3. For Linux systemd, add the values to the service override.

   [Service]
   Environment="OLLAMA_NUM_PARALLEL=2"
   Environment="OLLAMA_MAX_QUEUE=64"
   Environment="OLLAMA_MAX_LOADED_MODELS=1"

4. Reload and restart Ollama.

   $ sudo systemctl daemon-reload
   $ sudo systemctl restart ollama

   Related: How to manage the Ollama service

5. Run concurrent smoke requests from clients and watch loaded models.

   $ ollama ps
   NAME           ID              SIZE     PROCESSOR    CONTEXT    UNTIL
   gpt-oss:20b    17052f91a42e    12 GB    100% GPU     131072     5 minutes from now