Should AI agents get your personal data? A debate over trust

At the Masters of Scale Summit, moderator Jeff Berman asks whether people should stop trusting AI agents with their information. Signal president Meredith Whittaker argues for the proposition; Neil Chilson, head of AI policy at the Abundance Institute, argues against it.

Whittaker says Signal’s responsibility is to protect communications used in situations where privacy can be a matter of life and death. She wants technology that earns trust through verifiable properties rather than a promise from a company.

To arrange a birthday dinner, an agent might need contacts, payment information, a calendar, browser access and messages. Whittaker says the convenience depends on extensive permissions and on the agent performing multiple actions without asking at every step.

She identifies prompt injection as a serious weakness: text encountered by an agent can be treated as an instruction. She cites reported attacks on assistants and browsers, a test in which safeguards failed in some cases, and a calendar-based attack involving a smart-home device.

Chilson contrasts agents with the familiar internet bargain in which companies collect data to fund advertising. He argues that agents can work with a person’s data for that person’s purposes and make tasks once delegated mainly by wealthy people more accessible.

He says agents can also make privacy practices less burdensome: reading terms, using privacy tools or opting out of data brokers. He says he used an agent to request removal from five major data brokers with a short prompt.

Whittaker replies that consumer agents are often controlled by companies rather than their users. She cites reported cases of Meta’s Muse exposing an address or private information and argues that broad access breaks down the separation between applications.

She argues that an agent user’s choice can expose other people too. If someone gives an agent access to contacts and communications, a contact who never installed it may be affected; the consequences can be especially serious for human-rights workers.

Chilson agrees that agents need better security but says the comparison should be with an imperfect status quo. Whether to trust one depends on the agent, its task and the access requested. He says he generally allows agents to read online but not write.

Asked why convenience justifies prompt-injection risk, Chilson compares the problem to messaging: people did not abandon communication because some platforms were unsafe, but built better platforms. He thinks use and competition can help develop better agents.

Berman asks how an agent differs from a human assistant who has access to sensitive information. Whittaker says a human assistant is accountable through a working relationship and can be dismissed, whereas a failure by a broadly deployed system can harm many people.

Whittaker compares some agent architectures to targeted malware: extensive device permissions, screenshots or accessibility access, data processing and actions taken as the user. She says a bank blocking one agent illustrates concerns about distinguishing the customer’s intent from an injected command.

Chilson says an opportunity exists for companies to offer agents people can trust. In his view, a product acting on a customer’s behalf has an incentive to protect that relationship, even though companies and users will make mistakes.

Asked why Signal does not make an agent, Whittaker says Signal does not collect the data such a product would need and focuses on verifiable private messaging. She emphasizes that privacy can be a necessity rather than a luxury.

She questions the marketing promise of a universal assistant: the more an agent does across applications, the more information and access it may need. Confirmations and isolation can reduce exposure, she says, but may also constrain the seamless experience being promised.

Whittaker argues that screenshots or off-device processing can expose information after an encrypted application has displayed it. She says this shifts practical control from the application’s security boundary to the agent and its provider.

Chilson challenges the idea that an agent must receive complete access. He says permissions can be limited by task, and recommends testing incrementally and verifying behavior rather than treating every use as all-or-nothing.

Berman proposes a small experiment: allow an agent to summarize a separate newsletter inbox. Whittaker asks whether users know what information a product collects, citing reporting about contact and communication data that she says was not obvious in its marketing.

Chilson responds that an agent could help people read and understand the terms they click through. He says he was aware of Muse’s data use and considered it a worthwhile tradeoff, while accepting that others may decide differently.

In closing, Whittaker returns to the effect on third parties: a user cannot consent on someone else’s behalf to exposing their communications. She argues that operating-system-level access can undermine the privacy guarantees of individual applications.

Chilson agrees that sharing one person’s information can affect others, but says the problem is not unique to agents. His position is to improve the tools rather than disengage, including using them to make privacy protections such as data-broker opt-outs easier.