“Here’s how an order-1 model stacks up against GPT-2 with arithmetic coding in compressing a famous Charles Dickens quote: order-1 434 bits, 24% of original; GPT-2 176 bits, 10% of original.”
A language model and a compressor are doing the same job. Both assign probabilities to the next symbol, and a better guess buys fewer bits. The Dickens example makes it concrete: a trivial order-1 model gets to 24 percent, GPT-2 gets to 10. That also explains why these models memorize their training data. Compression is what they are for.