“One of the rules was an instruction not to leak the rules.”
Marvin von Hagen got GitHub Copilot Chat to dump its hidden prompt by claiming to be an OpenAI developer. That is the entire attack. Microsoft’s defence was a line in the prompt asking the model nicely to keep the prompt secret. Security by polite request fails the first time someone lies to it.