Petabyte Data Streaming & Worker Prefetching
As GPU compute speeds increase, data ingestion becomes the primary bottleneck in large-scale pre-training. An 8-GPU node training a vision-language model can consume over 40,000 samples per second, requiring high-throughput data streaming, on-the-fly tokenization, and ring-buffered prefetching.
Deterministic Shard Shuffling
Instead of centralizing the dataset index across 10,000 ranks (which crashes metadata servers), each rank deterministically computes its pseudo-random sample partition using seed permutations.
Multi-Process Ring Buffers
DataLoader worker processes stream chunks from cloud storage, decompress records, tokenize text, and populate a shared-memory ring buffer. The GPU simply pops pre-assembled tensors without waiting on disk I/O.
Ray & NVIDIA DALI Hardware Acceleration
Frameworks like Ray Data and NVIDIA DALI offload image decoding, JPEG decompression, and data augmentation directly to GPU hardware NVDEC/JPEG engines, bypassing host CPU cores.
If total batch ingest and decompression time exceeds the GPU compute execution time of forward + backward pass, GPUs enter an idle starve state, directly reducing Model FLOPs Utilization (MFU).
- Group small dataset files into large 100MB–500MB webdataset/tar shards to maximize sequential read throughput from object stores.
- Set num_workers = 4 to 8 per GPU and enable persistent_workers=True in PyTorch DataLoaders to avoid worker recreation overhead between epochs.
- Use local NVMe SSDs as a read-through cache for frequently accessed validation splits.