How to run parallel downloads with cURL

Running several HTTP transfers at the same time with cURL shortens completion time for bulk downloads, large data exports, and sharded datasets, especially on high-latency links. Multiple connections stay active together, so the network link is used more efficiently instead of waiting for each request to finish before starting the next.

Modern cURL builds accept multiple URLs in a single invocation and, from version 7.66.0 onward, can process them concurrently with the -Z or –parallel option. Internally the tool drives the libcurl multi interface, opening several connections, reusing sockets when possible, and reporting progress across all active transfers in one terminal window.

Concurrent transfers rely on a recent cURL version and on sensible concurrency limits that avoid overwhelming either side of the connection. The commands below assume Ubuntu with a repository-provided curl (for example 7.81.0 or newer), but the same flags apply on most Linux systems. Very aggressive parallelism can trigger throttling, consume API quotas quickly, or saturate local CPU and bandwidth, so moderate settings are usually safer than pushing the maximum.

1. Open a terminal where shell access is available.

   $ whoami
   alice

2. Confirm that the installed curl version supports parallel transfers.

   $ curl --version
   curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8
   Release-Date: 2022-01-05
   Protocols: file ftp ftps http https
   Features: AsynchDNS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM SSL TLS-SRP
   ##### snipped #####

   cURL version 7.66.0 or newer understands -Z and –parallel for concurrent transfers; older builds either ignore these options or exit with an error.

3. Create a configuration file that lists each url to download and an output directive for the destination file when needed.

   $ cat > urls.conf << 'EOF'
   url = "https://example.com/data-part1.bin"
   output = "data-part1.bin"
   
   url = "https://example.com/data-part2.bin"
   output = "data-part2.bin"
   
   url = "https://example.com/data-part3.bin"
   output = "data-part3.bin"
   EOF

   Options in the configuration file use the same syntax as /home/user/.curlrc, so headers, cookies, authentication tokens, and other per-request settings can be added alongside each url.

4. Run curl in parallel mode against the configuration file with a conservative concurrency limit.

   $ curl --parallel --parallel-max 3 --parallel-immediate --config urls.conf
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   [1/3] https://example.com/data-part1.bin
   [2/3] https://example.com/data-part2.bin
   [3/3] https://example.com/data-part3.bin
   ##### snipped #####

   Increasing –parallel-max too high can overload remote servers, exhaust API quotas, or congest local network links; the default upper limit without this option is 50 concurrent transfers.

5. Optionally run a small set of URLs in parallel directly on the command line without a configuration file.

   $ curl --parallel --parallel-max 3 \
     -O https://example.com/data-part1.bin \
     -O https://example.com/data-part2.bin \
     -O https://example.com/data-part3.bin

   The -O (remote name) flag saves each transfer using the file name from the URL, which keeps the invocation shorter when explicit output directives are unnecessary.

6. Verify that every expected file exists and has a non-zero size when the transfers finish.

   $ ls -lh data-part1.bin data-part2.bin data-part3.bin
   -rw-r--r-- 1 alice alice 100M May 10 10:10 data-part1.bin
   -rw-r--r-- 1 alice alice 100M May 10 10:10 data-part2.bin
   -rw-r--r-- 1 alice alice 100M May 10 10:10 data-part3.bin

   A successful batch usually shows a zero exit status from curl, progress lines without errors, and all target files present with plausible sizes.