“Concretely, Mixtral has 46.7B total parameters but only uses 12.9B parameters per token.”
Mixtral routes each token through two of eight expert blocks, so it runs at the speed and cost of a 12.9B model. Mistral claims it beats Llama 2 70B on most benchmarks with 6x faster inference, under Apache 2.0. The training data is described only as “the open Web”. The weights are open, and what went into them stays a secret.