NEAR - Sponsor Image NEAR - Confidential swaps across 35+ chains Friend & Sponsor Learn more
News

Study Finds AI Sellers Reading Customer Prompts

Cybersecurity firm Dragos planted unique secrets in about 2 million AI prompts and found breached providers, cheap marketplace sellers reading logs, and privacy flags that were not honored.

Study Finds AI Sellers Reading Customer Prompts

HD Moore, chief research officer at Dragos, an industrial cybersecurity firm, published a study Nov. 19 showing AI providers are keeping and reading prompts they're supposed to discard. His team hid unique fake links and passwords in about 2 million prompts from July to November, then watched who used them.

What's the Scoop?

  • The Sellers: On inference marketplaces, where the cheapest seller usually wins the request, some sellers simply turned on logging and read customers' prompts. The leaks followed specific sellers, not the marketplace as a whole.

  • The Ignored Flag: Providers ignored "zero data retention," the setting meant to stop them storing prompts. A Chinese AI lab opened links from about two-thirds of prompts marked that way, including some sent through U.S. resellers.

  • The Breaches: Hackers were also inside three hosting providers, none of which knew until Dragos told them.

  • The Fix: The findings make the case for verification, and several teams are already building it. Antseed lets buyers require sealed hardware that keeps even the machine's owner from reading the prompt, and already fingerprints sellers to catch swapped models. Anthropic open-sourced a prototype that proves which model produced an answer, and Pearl is building its own with Attestable. MEGA says it's building verification and prompt scrubbing into its inference market, per co-founder Shuyao Kong. Learn more here.

David Christopher

785 posts

David is a writer/analyst at Bankless. Prior to joining Bankless, he worked for a series of early-stage crypto startups and on grants from the Ethereum, Solana, and Urbit Foundations. He graduated from Skidmore College in New York. He currently lives in the Midwest and enjoys NFTs, but no longer participates in them.

No Responses