“By far the biggest open-weight model right now.”
2.8 trillion parameters, LatentMoE to compress the linear layers, attention residuals weighted by attention score, and RoPE dropped entirely for NoPE. The attention residuals cost about 4 percent in training and 2 percent at inference for better validation loss, which is the kind of tradeoff nobody publishes when they are selling a model. Everything here is efficiency work, not a new idea about how models should be built.