one of the often overlooked factors behind deepseek’s success lies...

built on this foundation haiscale ddp (which is another key innovation!) became deepseek’s specialized distributed training framework.
communication backend: haiscale ddp employs hfreduce instead of pytorch ddp’s default nccl.
computation-communication overlap: during backpropagation it performs asynchronous all-reduce on gradients allowing communication to overlap with computation.
performance advantage: since hfreduce avoids using gpu streaming multiprocessors for reductions the all-reduce remains fully asynchronous (preventing interference with gpu workloads). as a result training vgg16 with haiscale ddp took half the time compared to torch ddp (nccl backend) and achieved ~88 % parallel scalability when scaling from 32 to 512 gpus.
the fire-flyer ai-hpc paper has great details about these infrastructure nuances. highly recommended read for anyone interested in ml systems, distributed training and high-performance infrastructure design.



