-Content by TechnologyNewswire-
YouGov, the international research and data analytics group, today published a white paper on one of the most transparent tests yet of one-to-one human-to-digital-twin simulation.
Rather than comparing aggregate results against public datasets, the groundbreaking research checked each AI twin individually against its own human.
Working with OriginalVoices, an AI platform specializing in simulation, YouGov built digital twins; LLM-based simulations of real people’s attitudes, which it calls “E-twins” of 1,008 government ID-verified members of its proprietary panel. It then put the same new questions to both the E-twin and the real human respondent.
Fully 69% of the panelists responded within 48 hours, giving a direct check on the accuracy of their E-twins. In day-to-day operations, this validation can be done in as little as an hour.
The Key Findings From Yougov’s AI Twins Testing
- Rich Data Drives Accuracy – Adding panelists’ ratings of 4,000+ entities lifted person-level accuracy (within one point on a five-point scale) from 78% to 90%
- YouGov’s E-twins Answered What They Were Never Taught – Fiction books were not part of the training data, but when asked about them, the E-twins still came within one point of their human 90% of the time
- Twin Confidence Tracks With Accuracy – Each open-ended answer a digital twin gives is assigned a confidence level (High, Medium or Low) by OriginalVoices. Panelists said the answer described their own views “very” or “somewhat” well for 81% of High-confidence answers, 72% of Medium and 67% of Low, giving researchers a practical guide to which individual twin responses to trust
- YouGov’s Real Panelists Regularly Identify with their Twins’ Opinions – 75% of the time, panelists judged their E-twin’s open-ended answers as describing their own views
- Less stereotyping – More contextual training data reduced twins’ reliance on demographics, with no pattern of biased performance by age, gender, race or education
More from Artificial Intelligence
- Disney Is Teaching AI Its Creative Style – What Does This Mean For The Future Of Entertainment?
- Your Patient Records Could Become AI Training Data – What Do Privacy Rules Say About De-Identified Health Data?
- Google Says AI Could Help Make Software “Unhackable”, But Is That Realistic?
- Experts Comment: Was Last Week’s Tech Stock Crash A Warning Sign For Investors
- Nobody Agrees On What AI Is Doing To The Workforce (And That’s What’s Causing The Real Problems)
- Mercedes-Benz Is Moving Small AI Models Into Its Cars – How Much Of The Cloud’s Work Can They Take On?
- What Is Model Context Protocol, And Why Is It Becoming So Important For AI?
- The Sandbox Fights Back: Joe Security Launches Adaptive Detonation
Current Simulations Are Built On Shaky Foundations
YouGov says much of the evidence offered to date by simulators across the industry rests on publicly available datasets. Those questions may already sit in the training data of the underlying language models, so a twin can appear to know an answer it has simply absorbed and respondents cannot be recontacted to verify the results.
Aggregate comparisons can also mask twins that flatten towards the average or stereotype.
Simulated Data Providers Need to Show Their Workings
YouGov is calling on the industry to develop shared standards of verification and validation for defined use cases.
It challenges all simulators to follow its lead in validating twins against their individual human, test them on unseen topics, publish their methods, show their misses and predict real-world outcomes in public.
Stephan Shakespeare, CEO and co-founder of YouGov: “The research industry is being transformed by AI in many ways but especially when it comes to simulation. Whether this is a dangerous development or a beneficial advance depends on whether clients can know if its outputs are reliable. I challenge everybody in the industry to work together to establish standards for verifying and validating simulated data. This white paper shows what we think these standards should involve, including the ability to validate simultaneously. If the industry wants simulation to be trusted by clients and the public, it needs to earn it and this can only be achieved through widely agreed standards.”
YouGov’s white paper also publishes the gaps future analysis will address. The current study did not measure a ceiling; how often a person repeats their own answer to the same question. A twin can’t reasonably beat the person it copies: if panelists repeated their own rating 70% of the time, the twins’ 58% exact-match rate would represent 83% of that ceiling. YouGov will test this in the next iteration of its research.
-This is a paid press release published via CyberNewswire-
