“Here’s a number of improvements that we applied in order to reduce 75th percentile of end-to-end latencies for single line completion from 1.8 seconds to under 900 milliseconds over the past months”
Cody autocomplete started as a prompt to Claude Instant and a lot of frustrated users. The model kept repeating code already on the next line and writing function after function nobody asked for. Sourcegraph fixed it with rule-based guards, indentation checks and finally a switch to StarCoder. The magic AI assistant is mostly heuristics bolted on to stop the model from doing what models do.