one of the often overlooked factors behind deepseek’s success lies...

@novasarc01
λux@novasarc01
19 views Oct 28, 2025 ~1 min read
Advertisement
1
one of the often overlooked factors behind deepseek’s success lies in how the team built their distributed infrastructure from the ground up (despite severe gpu constraints!). their custom communication library hfreduce replaced nccl and delivered substantially higher bandwidth efficiency. while nccl’s all-reduce requires gpu kernel execution (interfering with concurrent computational kernels) hfreduce was designed specifically to overcome such bottlenecks on the fire-flyer 2 cluster. it drastically reduced pcie bidirectional bandwidth usage relative to nccl’s ring topology by offloading the reduction to the cpu and routing data via infiniband and host memory thereby bypassing the epyc rome cpu limitation.

built on this foundation haiscale ddp (which is another key innovation!) became deepseek’s specialized distributed training framework.

communication backend: haiscale ddp employs hfreduce instead of pytorch ddp’s default nccl.

computation-communication overlap: during backpropagation it performs asynchronous all-reduce on gradients allowing communication to overlap with computation.

performance advantage: since hfreduce avoids using gpu streaming multiprocessors for reductions the all-reduce remains fully asynchronous (preventing interference with gpu workloads). as a result training vgg16 with haiscale ddp took half the time compared to torch ddp (nccl backend) and achieved ~88 % parallel scalability when scaling from 32 to 512 gpus.

the fire-flyer ai-hpc paper has great details about these infrastructure nuances. highly recommended read for anyone interested in ml systems, distributed training and high-performance infrastructure design.
Media image
Media image
Media image
Media image
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement