How to fix CUDA out of memory errors in PyTorch

CUDA out of memory errors mean the selected NVIDIA GPU cannot satisfy another allocation from the running PyTorch process. The failure usually appears during a forward or backward pass, but a later batch can fail after earlier batches succeed when a shape, sequence length, or retained tensor makes one iteration larger than the rest.

PyTorch uses a CUDA caching allocator, so nvidia-smi can show memory as used even after tensors were freed for reuse inside the same process. torch.cuda.memory_allocated() reports live tensor memory, torch.cuda.memory_reserved() reports memory held by the allocator, and torch.cuda.empty_cache() cannot free tensors that Python still references.

The original failing command supplies the baseline for every retry, while the memory report separates external GPU pressure from memory held by the current process. Reducing per-step memory comes before allocator settings; PYTORCH_ALLOC_CONF helps only after the report points to allocation fragmentation or changing batch shapes, and it does not make an oversized live tensor workload fit.

1. Reproduce the CUDA memory failure with the original per-device batch size.

   $ python train.py --batch-size 64
   torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.20 GiB. GPU 0 has a total capacity of 23.64 GiB of which 820.00 MiB is free.
   ##### snipped #####

   If torch.cuda.is_available() is false, or the error says no CUDA device is present, fix GPU detection before changing batch size.
   Related: How to fix PyTorch not detecting a GPU

2. List processes that already hold GPU memory.

   $ nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
   pid, process_name, used_gpu_memory [MiB]
   24810, python, 11264 MiB

   If another process owns the missing memory, reschedule the job, select a free GPU, or confirm ownership before stopping it; an unknown GPU process may be another user's training run or an active service.

3. Add a temporary CUDA memory report around the failing training or inference call.

   train.py

   def gib(value):
       return value / 1024 ** 3
    
   try:
       train_one_epoch(model, loader, optimizer)
   except torch.cuda.OutOfMemoryError:
       print(f"allocated={gib(torch.cuda.memory_allocated()):.2f} GiB")
       print(f"reserved={gib(torch.cuda.memory_reserved()):.2f} GiB")
       print(f"peak_allocated={gib(torch.cuda.max_memory_allocated()):.2f} GiB")
       print(torch.cuda.memory_summary(abbreviated=True))
       raise

   Replace train_one_epoch(…) with the smallest block that still reproduces the failure, such as one validation pass, one training step, or one model call.

4. Rerun the failing job and read the reported CUDA memory numbers.

   $ python train.py --batch-size 64
   allocated=21.92 GiB
   reserved=22.50 GiB
   peak_allocated=22.18 GiB
   torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.20 GiB.
   ##### snipped #####

   When allocated is close to the GPU capacity, live tensors or activations are the pressure point. When reserved is much larger than allocated, allocator fragmentation or changing allocation sizes may also be involved.

5. Retry with a smaller per-device batch size from a fresh Python process.

   $ python train.py --batch-size 8
   epoch=1 batch=40 loss=1.184 peak_allocated=7.82 GiB
   completed epoch=1

   Lower the per-device batch first; if the dataset contains variable sequence lengths or image sizes, keep margin for the largest batch instead of tuning only for the first successful batch.

6. Preserve the effective training batch with gradient accumulation when optimizer updates still need the original batch size.

   $ python train.py --batch-size 8 --grad-accum-steps 8
   epoch=1 micro_batch=8 grad_accum_steps=8 effective_batch=64
   peak_allocated=7.90 GiB
   completed epoch=1

   Use the option names from your training entry point. The important change is a smaller microbatch on the GPU and fewer optimizer steps per full effective batch.
   Related: How to run gradient accumulation in PyTorch

7. Enable automatic mixed precision when the model and loss are compatible with lower-precision CUDA operations.

   train.py

   scaler = torch.amp.GradScaler("cuda")
    
   for inputs, targets in loader:
       optimizer.zero_grad(set_to_none=True)
       with torch.amp.autocast("cuda"):
           loss = loss_fn(model(inputs), targets)
       scaler.scale(loss).backward()
       scaler.step(optimizer)
       scaler.update()

   If loss values become NaN or gradients overflow after enabling AMP, disable it for that model path and keep the batch-size or checkpointing fix instead.
   Related: How to enable mixed precision in PyTorch

8. Checkpoint a memory-heavy model block when activations still dominate the peak allocation.

   train.py

   from torch.utils.checkpoint import checkpoint
    
   x = checkpoint(model.encoder_block, x, use_reentrant=False)

   Activation checkpointing trades extra forward computation during backward for lower saved-activation memory. Wrap deterministic blocks first, and avoid moving tensors to a new device inside the checkpointed function.

9. Use allocator expansion only when the memory report points to fragmentation or changing batch shapes.

   $ PYTORCH_ALLOC_CONF=expandable_segments:True python train.py --batch-size 8 --grad-accum-steps 8
   epoch=1 micro_batch=8 grad_accum_steps=8 effective_batch=64
   peak_allocated=7.88 GiB
   completed epoch=1

   Set PYTORCH_ALLOC_CONF before the Python process starts. Older examples may use PYTORCH_CUDA_ALLOC_CONF, which remains a compatibility alias, but allocator tuning cannot free tensors that are still live.

10. Remove the temporary memory-report block after the selected retry completes.

    Keep the successful batch size, accumulation count, AMP setting, checkpointed block, or allocator setting in version control so the next run uses the same memory boundary.