\
We are experiencing a conflict between the economic pressure to produce rapid and cheap research results and the scientific demand for rigor. Hundreds, if not thousands, of realistic characters can be generated in minutes from providers who promise excellent results. But these often function as methodological black boxes, producing results that cannot be validated, may contain hidden biases, and may silently mislead decision-making.
The synthetic data market is growing rapidly, with valuations expected to rise from approximately $267 million in 2023 to over $4.6 billion by 2032. Driven by the demand for instant insights in an always-on economy, 95% of leaders have insight plans to use synthetic data within the next year, and the appeal is clear. Speed, scalability, cost efficiency and the ability to generate insights from niche audiences are key factors.
To transform synthetic testing from a purely experimental approach to a reliable and scalable practice, organizations must address these risks directly. Different approaches can help overcome skepticism and create a more sustainable model. It is important to identify key problem areas and address them directly.
While cost savings and speed of insight acquisition are compelling reasons for adoption, several challenges remain. The most successful organizations understand the strengths and weaknesses of different synthetic tools and when to use them.
Your customers search everywhere. Make sure your branding introduces himself.
The SEO toolkit you know, plus the AI visibility data you need.
Start your free trial
Start with
Common challenges with synthetic search approaches
Why general LLMs fail to meet expectations
Why can’t you just ask ChatGPT your research questions? A common misconception in synthetic research is that providing an LLM with a detailed backstory guarantees a representative outcome. Recent large-scale experiments suggest otherwise.
Initial studies show that pushing an LLM like ChatGPT, Claude or Gemini to produce more content per person increases bias/homogeneity instead of creating a diverse set of results. For example, the characters used to predict the results of the 2024 US presidential election (with detailed backstory provided by an LLM) won every state for the Democrats and failed to reflect the political diversity of the population.
This phenomenon highlights a problem known as bias recycling, a pervasive problem in artificial intelligence that affects everything from facial recognition to synthetic research, as LLMs are trained on Internet data that disproportionately reflects a Western, Educated, Industrialized, Wealthy, Democratic (WEIRD) worldview. Asking models to be different characters produces a statistical average filtered through this bias, recycling the exclusion as AI neutrality.
Additionally, synthetic respondents may suffer from the Pollyanna Principle, which is the tendency for LLMs to be overly pleasant and positive in their responses to user suggestions. Most users of generative AI chat interfaces have probably encountered this: Ideas are greeted with encouragement as a “great idea” or “good choice” rather than objective evaluation.
For example, in a Usability testing that compares synthetic respondents to human respondentssynthetic users reported completing all courses online. Where human users might report dropping out of most online courses, synthetic users reported completion.
The high dropout rates among real users confirmed that the synthetic respondents were trying to say what they thought the experimenters wanted to hear. This servility can lead to the assertion of poor product concepts by helpful AI agents.
Fine-tuning provides the context that synthetic approaches lack
Aren’t LLMs trained on a broad enough set of information to produce realistic use cases in almost any scenario? The most effective way to align synthetic respondents with reality is to refine them using first-party data. While general LLMs provide decent baseline estimates for existing products, they are facing new challenges and underrepresented segments.
In an experimentone team interrogated a basic GPT model for a fictional pancake-flavored toothpaste and stumbled straight on to the Pollyanna principle. Without training data, the model predicted that people would like it – in other words, it hallucinated a preference for novelty. Once the researchers tuned the model on data from previous surveys of toothpaste preferences, the output correctly changed to negative.
In another study on the desirability of a built-in projector in laptops, the base model overestimated willingness to pay by a factor of three. After refining the survey data on standard laptops, the error was corrected, aligning the synthetic results with human benchmarks.
Get the best results with synthetic
The competitive advantage in synthetic research is not the model itself – which is becoming a commodity – but the proprietary context that conditions it. For example, Dollar Shave Club used synthetic panels powered by categorical data to validate new customer segments in days rather than months, achieving results that mirrored human behavior with a fraction of the effort.
Some approaches can help you get the best results from synthetic search.
Train synthetic, test real
To address some of these challenges, the market research industry has proposed an industry-wide validation methodology known as train-synthetic, test-real (TSTR). In this approach, models are trained on synthetic data and tested for predictive validity against a sample of real-world data. The first results have been positive.
In research Led by Stanford University and Google DeepMind, digital agents trained on interview data replicated human survey responses with 85% accuracy and social forces with 98% correlation.
This approach recognizes the disadvantages of relying solely on standardized LLMs as a starting point, as well as the risks of taking synthetic results at face value without validation. By using synthetic methods upfront and validating with real data, teams can realize time and cost savings while building confidence in the results.
Using governance and transparency
Succeeding with synthetic research means that researchers and readers cannot embrace the synthetic person fallacy: the belief that LLMs possess the equivalent of human psychology and personality traits.
Instead, a more rigorous validation approach is needed, supported by governance controls, well-documented processes and transparency in the methods used.
A person transparency checklist can guide researchers as they interact with synthetic characters:
- Application domain: The specific task that the person is intended to perform.
- Target population: The demographic target group the person intends to represent, rather than relying on generic descriptions.
- Data provenance: Whether existing datasets were reused or modified to build the characters.
- Ecological validity: Whether experimental interaction reflects real-world usage contexts.
Transparency solves two challenges. It addresses ethical concerns around disclosure and builds trust by showing how synthetic approaches work and where they fail. As synthetic influence grows, the distinction between real and synthetic content will become critical.
Trust but verify
A realist approach to synthetic research means abandoning the belief that LLMs inherently reflect human psychology and instead focusing on empirical benchmarking, fine-tuning and transparency.
Synthetic research works if you respect its limits
Synthetic research shows great potential but is a promise with a grip. The promise is unprecedented speed and reach, while the problem is the risk of bias and hallucinations.
Recognizing these challenges and building governance and barriers to mitigate them will help you succeed. This also transforms internal skepticism into a structured governance approach that balances efficiency with results, creating a win-win.
