How to run parallel downloads with cURL

Running several downloads at the same time with cURL shortens artifact staging, mirror pulls, and bulk file collection because the network can stay busy while more than one transfer is in flight.

Without extra flags, cURL handles multiple URLs one by one. --parallel starts them together, --parallel-max limits how many transfers can run at once, and --remote-name-all with --output-dir stores each response under its remote filename in one destination directory. By default, cURL waits briefly to see whether later transfers can reuse or multiplex an existing connection; --parallel-immediate changes that preference when startup speed matters more than keeping the connection count low.

Parallel downloads still need predictable filenames, a conservative concurrency limit, and failure handling that cannot hide an earlier bad transfer behind a later successful one. Use a clean target directory or names that are safe to overwrite, keep the batch small for shared or rate-limited endpoints, combine --fail with --fail-early for HTTP errors, and add --remove-on-error so failed transfers do not leave partial files behind. cURL 8.16.0 and later builds also add --parallel-max-host for per-host caps when one server should have a stricter limit than the whole batch.

1. Create a destination directory for the files that cURL will save.

   $ mkdir -p downloads

   --output-dir writes every downloaded file into this directory and the transfer fails if the directory does not already exist.

2. Start the batch with --parallel, keep the concurrency limit small, and save each URL by its remote filename.

   $ curl --parallel \
     --parallel-max 3 \
     --remote-name-all \
     --output-dir downloads \
     --fail --fail-early --show-error \
     --remove-on-error \
     https://downloads.example.net/notes.txt \
     https://downloads.example.net/amd64.tgz \
     https://downloads.example.net/arm64.tgz

   --remote-name-all applies the remote filename rule to every URL. --fail-early makes a failed transfer fail the batch even when other downloads succeed, and --remove-on-error deletes that transfer's incomplete target file. If the local build rejects --remove-on-error, remove partial files manually after any failed transfer. The hero image above shows the parallel progress meter from this same command.

3. Confirm that every expected file exists in the output directory before you use the batch output.

   $ ls -lh \
     downloads/amd64.tgz \
     downloads/arm64.tgz \
     downloads/notes.txt
   -rw-r--r-- 1 user user 26K Jun  6 10:11 downloads/amd64.tgz
   -rw-r--r-- 1 user user 28K Jun  6 10:11 downloads/arm64.tgz
   -rw-r--r-- 1 user user  24 Jun  6 10:11 downloads/notes.txt

   Missing files, zero-byte files, or unexpectedly small sizes mean one or more transfers failed. Rerun only the affected URLs instead of repeating a successful batch.