Salesforce study finds LLM agents flunk CRM and confidentiality tests

“Using the benchmark tool CRMArena-Pro, the team also showed performance of LLM agents drops to 35 percent when a task requires multiple steps.”

Salesforce’s own researchers found LLM agents finish 58% of single-step CRM tasks and 35% of multi-step ones. They also leak confidential data unless prompted not to, and that prompting makes them worse at the task. Benioff pitched agents to investors as a very high margin opportunity. His research team just measured the margin of error.