“This release marks the release of the strongest linear transformer (in terms of eval benchmarks) to date.”
RWKV trained 7.52 billion parameters on 1.1 trillion tokens and claims 10-100x cheaper inference than a transformer. On English it trails Mistral and only trades blows with MPT-7B. The 100+ language claim rests on benchmarks covering 23 languages. “Soaring past” means catching up to last year’s models.