“For under $20 USD, our attack extracts the entire projection matrix of OpenAI’s Ada and Babbage language models.”
OpenAI keeps its model sizes secret and calls that safety. Carlini and 13 co-authors pulled the hidden dimension of Ada and Babbage, 1024 and 2048, through the normal API for under $20. They also recovered the exact hidden dimension of gpt-3.5-turbo and put the full matrix at under $2,000 in queries. The secrecy was only as deep as the API bill.