Hy4 preview went from 1.5TB to 214GB. The trick: give ZERO a job
Hy4 preview's weights dropped from 1.5TB to 214GB. Here's the part that got skipped：
Why 1.25 bits
The idea behind Sherry is old. Ternary quantization turns every weight into -1, 0, or +1, so a multiply becomes an add. That's exactly what CPUs and cheap edge chips want. The hard part was always the packaging.
Hardware counts in powers of two, and three values don't fit. For a decade everyone treated the zero as the price of going ternary: dead weight sitting inside the weights.
Sherry flipped that. Instead of paying for the zero, it put the zero to work. Force every group of four weights to hold exactly one zero, and the zero's position becomes information. Four slots, three signs: 32 states, which fit in exactly five bits. That's 1.25 bits per weight with nothing wasted, and the four-way grouping lines up with how hardware lanes actually run.
The arithmetic fits in one line. The move is realizing the zero could do a job.
Why the model doesn't collapse
The launch diagram skips a second problem. Force that 3:4 sparsity during training and call it done, and the weights drift toward binary, losing the ternary range you paid for.
We traced it to weight trapping. The sparsity homogenizes the gradients, so updates start looking alike, and the model quietly turns 1-bit instead of ternary.
The fix is Arenas, a full-precision residual added during training and annealed to zero. It breaks the gradient symmetry and lets the weights keep their ternary diversity. When training ends, the residual is gone. Zero overhead at inference. Only the sparse ternary weights remain.
Why it lands on Hy4 preview
The last reason is context. A MoE like Hy4preview activates only 49B of its 770B parameters per token, about 6.4%, routing through 8 of 256 experts plus one shared.
That redundancy is exactly where aggressive quantization hurts least. The experts are so over-provisioned that you can push the compression hard there and leave the sensitive layers wide. That's the bet, and it's why this isn't just a paper trick.
Shrinking the model was the easy part. Giving the zero something to do was the hard part.
Sherry is the method. Hy4preview is it running on a real 770B MoE.
Paper: arxiv.org/abs/2601.07892
Code: github.com/Tencent/AngelSlim