“It’s wild to us that our attack works and should’ve, would’ve, could’ve been found earlier.”
Ask ChatGPT to repeat the word “poem” forever and it eventually spits out memorized training data, including a real email address and phone number. About $200 of queries pulled several megabytes, and the researchers estimate a gigabyte is reachable. Banning the word trick patches the exploit. The model still memorized your phone number.