“One fundamental issue we ran into was that the attention layers in Transformers scale quadratically in sequence length: going from 32k length to 64k length isn’t 2x as expensive, but 4x more expensive.”
Stanford’s Hazy Research lab wants to replace attention with convolutions that scale at N log N. Its H3 models beat GPT-Neo on the Pile at 1.3B and 2.7B parameters, but still kept two attention layers. The Hyena successor drops them and runs slightly slower than a Transformer at 2K tokens. The whole bet rests on the FFT, which modern GPUs barely support.