“With this patch, we’re now neck-and-neck with llama.cpp for long-context generations - at 58.7 tok/s vs. 58.8 tok/s - while handily beating it for short contexts at 63.8 tok/s vs. 61.0 tok/s.”
A naive CPU build of Mistral-7B ran at 0.6 tokens per second. Hand-written CUDA on one RTX 4090 got it to 63.8, past llama.cpp on short contexts. Matrix multiplies are 94.4% of kernel time, and one attention kernel used only 8% of memory throughput before the rewrite. Inference is a memory bandwidth problem, and most of the AI stack is wrapping paper around it.