Study Finds AI Sellers Reading Customer Prompts
Cybersecurity firm Dragos planted unique secrets in about 2 million AI prompts and found breached providers, cheap marketplace sellers reading logs, and privacy flags that were not honored.
HD Moore, chief research officer at Dragos, an industrial cybersecurity firm, published a study Nov. 19 showing AI providers are keeping and reading prompts they're supposed to discard. His team hid unique fake links and passwords in about 2 million prompts from July to November, then watched who used them.
What's the Scoop?
The Sellers: On inference marketplaces, where the cheapest seller usually wins the request, some sellers simply turned on logging and read customers' prompts. The leaks followed specific sellers, not the marketplace as a whole.
The Ignored Flag: Providers ignored "zero data retention," the setting meant to stop them storing prompts. A Chinese AI lab opened links from about two-thirds of prompts marked that way, including some sent through U.S. resellers.
The Breaches: Hackers were also inside three hosting providers, none of which knew until Dragos told them.
The Fix: The findings make the case for verification, and several teams are already building it. Antseed lets buyers require sealed hardware that keeps even the machine's owner from reading the prompt, and already fingerprints sellers to catch swapped models. Anthropic open-sourced a prototype that proves which model produced an answer, and Pearl is building its own with Attestable. MEGA says it's building verification and prompt scrubbing into its inference market, per co-founder Shuyao Kong. Learn more here.