Tldr: we feed more data into llms and it trains faster, then we resume as usual. Very easy to reproduce, significantly better loss at equal flops, imo worth an A/B if you pretrain models :)

Creator 0x707d…ce6c·3.90% creator tax