“From my calculations based on sizes of each layer, Apple’s predictive text model appears to have about 34 million parameters, and it has a hidden size of 512 units.”
Apple announced “a Transformer language model” at WWDC, and the model behind it has about 34 million parameters. That is smaller than the smallest GPT-2, and its layers carry names like gpt2_transformer_layer_3d. Apple pitched a language model. Users get a GPT-2 knockoff that mostly suggests one word at a time.