What Interview-Based AI Agents Can Predict About People

An earlier Stanford-associated paper explored generative agents living in a simulated town, forming memories, relationships and plans. I was struck by the possibility of more dynamic video-game characters and simulated societies. A new paper by Joon Sung Park and collaborators asks a different question: can agents informed by interviews approximate specific people’s answers and decisions?

The researchers interviewed 1,052 US participants and constructed agents from their accounts. They compared simulated responses with people’s answers on surveys and behavioral tasks. Calling the agents personality clones overstates what this establishes: the study measures particular responses and experimental outcomes, not whether an entire human personality has been reproduced.

The headline result concerns the General Social Survey (GSS). The paper reports an average normalized accuracy of 0.85: agent prediction accuracy is compared with how consistently participants repeated their own answers two weeks later. The 2024 paper reports 68.85% raw agent accuracy and 81.25% participant retest accuracy; 0.85 is a relative measure, not 85% of a person or 85% raw accuracy.

The evaluation also includes the Big Five personality inventory, behavioral economic games and social science experiments. These are separate tests with different metrics, so the GSS figure cannot be applied to all of them. The researchers describe possible uses in piloting interventions and investigating institutions and networks.

I imagine using a simulated population to explore how people might respond to a new tax policy before implementing it. That is a possible future application, not something demonstrated by this study. Accurate survey predictions alone do not establish that a simulated society can reliably predict the real-world effects of a policy.

The study combines a language model with approximately two-hour qualitative interviews. It evaluates the agents against the GSS, Big Five, economic games such as the dictator and public goods games, and five selected social science experiments. Participants also repeated the assessment after two weeks to provide a measure of their own consistency.

The interviews are semi-structured: a common script is supplemented with follow-up questions tailored to the participant’s answers. An AI interviewer conducts the voice conversation. This lets people elaborate on what matters to them rather than only choosing from fixed survey answers.

The recorded interview is transcribed and supplied to the agent as its memory. The earlier generative-agent work used brief backstories and memories formed through interaction; here, researchers use a real person’s interview transcript as context. The agent is prompted to imitate that participant, and its responses are compared with the participant’s actual answers.

Interview topics range from life history, education, family and relationships to views on social issues. The interviewer generates follow-up questions based on each answer. For multi-step experiments, the agent can also receive short descriptions of previous stimuli and its own earlier responses.

On the GSS, interview-based agents achieved 0.85 normalized accuracy, compared with 0.71 for demographic-based agents and 0.70 for persona-based agents in the paper. I take this as evidence that richer individual information helped on this evaluation; it does not mean demographic bias has disappeared or that the agents understand people completely.

On the Big Five, the reported normalized correlation is 0.80. For the economic games, it is 0.66. These are correlations normalized against participant retest consistency, not percentages of decisions correctly predicted; the paper also reports no significant difference in mean absolute error between agent types for the economic games.

The authors tested how much interview material mattered by randomly removing 80% of the transcript, equivalent to about 96 minutes of a 120-minute interview. The remaining interview-based agents still reached 0.79 normalized GSS accuracy and outperformed a composite baseline built from other participant data in that analysis.

In another test, GPT-4o converted interviews into bullet-point summaries that removed the original wording while retaining factual content. These summary-based agents also performed well, though the paper reports a slightly lower normalized GSS accuracy of 0.83 than the full-interview agents. The comparison suggests the information in the interviews contributes to performance, without proving linguistic cues play no role.

The researchers also examined differences in prediction performance across political ideology, race and gender. For GSS political-ideology subgroups, they report that the gap between the best- and worst-performing groups fell from 12.35 percentage points with demographic-based agents to 7.85 with interview-based agents. This is a particular measured disparity, not proof that all biases were removed.

I wonder whether interviews at much larger scale could inform models and generate useful synthetic data. That is a speculative extension of the work. The paper’s measured results concern its recruited participants and selected assessments, so any broader use would need further validation and attention to the people represented.