How to debug DataLoader worker errors in PyTorch

PyTorch DataLoader worker failures often appear after the first few batches, when dataset code starts running inside subprocesses instead of the main training process. The top error may only say that an exception was caught in a DataLoader worker, while the line that names the missing file, bad transform, or invalid sample is lower in the traceback.

Keep the same dataset, batch size, sampler, and collate function, then set num_workers=0 so PyTorch loads data in the main process. PyTorch documents num_workers=0 as main-process loading, and worker subprocesses as the path used when the value is greater than zero.

After the direct traceback names the failing getitem(), collate_fn, or input record, repair that narrow layer before turning workers back on. If the corrected loader still fails only with workers, check process-safe code next, especially top-level dataset and collate definitions, a if __name__ == "__main__" guard on spawn-based platforms, and CPU tensors returned from dataset workers.

1. Create a reduced DataLoader reproducer that keeps the failing worker count and dataset access pattern.

   debug_dataloader_worker.py

   from argparse import ArgumentParser
    
   import torch
   from torch.utils.data import DataLoader, Dataset
    
    
   class CustomerDataset(Dataset):
       def __init__(self, state):
           self.records = [
               ("data/customer_001.pt", [0.10, 0.20, 0.30], 0),
               ("data/customer_002.pt", [0.60, 0.40, 0.90], 1),
               ("data/customer_003.pt", None, 1),
               ("data/customer_004.pt", [0.20, 0.70, 0.50], 0),
           ]
           if state == "fixed":
               self.records[2] = ("data/customer_003.pt", [0.55, 0.80, 0.35], 1)
    
       def __len__(self):
           return len(self.records)
    
       def __getitem__(self, index):
           path, values, label = self.records[index]
           if values is None:
               raise FileNotFoundError(f"missing training sample: {path}")
           return torch.tensor(values, dtype=torch.float32), torch.tensor(label)
    
    
   def main():
       parser = ArgumentParser()
       parser.add_argument("--workers", type=int, default=2)
       parser.add_argument("--state", choices=["broken", "fixed"], default="broken")
       args = parser.parse_args()
    
       loader = DataLoader(
           CustomerDataset(args.state),
           batch_size=2,
           shuffle=False,
           num_workers=args.workers,
       )
    
       batch_count = 0
       for batch_features, batch_labels in loader:
           batch_count += 1
           print(
               f"batch {batch_count}: "
               f"features={tuple(batch_features.shape)} "
               f"labels={batch_labels.tolist()}"
           )
       print(f"completed batches: {batch_count}")
    
    
   if __name__ == "__main__":
       main()

   Keep the reproducer small, but preserve the loader options that affect the failure, especially batch_size, sampler, collate_fn, worker_init_fn, and num_workers. The dataset class and entry point stay at module scope so the script also works on spawn-based multiprocessing platforms.

2. Reproduce the failure with worker processes enabled.

   $ python debug_dataloader_worker.py --workers 2 --state broken
   batch 1: features=(2, 3) labels=[0, 1]
   Traceback (most recent call last):
     File "debug_dataloader_worker.py", line 53, in <module>
       main()
   ##### snipped #####
   FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 1.
   Original Traceback (most recent call last):
   ##### snipped #####
     File "debug_dataloader_worker.py", line 24, in __getitem__
       raise FileNotFoundError(f"missing training sample: {path}")
   FileNotFoundError: missing training sample: data/customer_003.pt

   The worker wrapper confirms that the exception came from a subprocess. The lower Original Traceback block points back to the dataset method and record that raised the original error.

3. Run the same loader in the main process.

   $ python debug_dataloader_worker.py --workers 0 --state broken
   batch 1: features=(2, 3) labels=[0, 1]
   Traceback (most recent call last):
     File "debug_dataloader_worker.py", line 53, in <module>
       main()
   ##### snipped #####
     File "debug_dataloader_worker.py", line 24, in __getitem__
       raise FileNotFoundError(f"missing training sample: {path}")
   FileNotFoundError: missing training sample: data/customer_003.pt

   num_workers=0 disables worker subprocesses, so the dataset exception is raised directly in the training process. Use it for diagnosis, not as the final performance setting unless single-process loading is acceptable.

4. Fix the dataset path, parser branch, transform, or record named by the direct traceback.

   Use --state fixed in the reduced script to represent the corrected project record. In a real dataset, make the equivalent correction in the manifest, file lookup, transform, or getitem() branch that raised the exception.

5. Recheck the corrected dataset in the main process.

   $ python debug_dataloader_worker.py --workers 0 --state fixed
   batch 1: features=(2, 3) labels=[0, 1]
   batch 2: features=(2, 3) labels=[1, 0]
   completed batches: 2

6. Re-enable worker processes and retest the corrected loader.

   $ python debug_dataloader_worker.py --workers 2 --state fixed
   batch 1: features=(2, 3) labels=[0, 1]
   batch 2: features=(2, 3) labels=[1, 0]
   completed batches: 2

   If the loader now passes with num_workers=0 but still fails with workers, inspect multiprocessing-only causes: unpicklable dataset state, nested collate_fn or worker_init_fn functions, missing if __name__ == "__main__" protection on spawn-based platforms, or CUDA tensors being returned from worker processes.

7. Remove the temporary reproducer after the training dataset is fixed.

   $ rm debug_dataloader_worker.py