“It’s been trained on 525 billion tokens of, of code all permissively licensed code.”
Replit’s replit-code-v1-3b has 2.7 billion parameters and beats Codex on HumanEval, but only after finetuning on Replit’s own data. Its other benchmark is AmjadEval, named after Replit’s CEO. Shabani admits one model with great HumanEval scores failed the vibes test. When the benchmark is the boss’s gut, every release passes.