What to ask any synthetic research vendor

Pervasive Insights™ is a synthetic research panel built on a client's own human research: a searchable library of every study the client has run, and synthetic respondents calibrated to how that client's real customers answered, with the error reported topic by topic. It is a City Research Solutions product. City Research has run consumer research since 1979.

Nine questions worth putting to any vendor selling synthetic respondents, including this one. Each is followed by how Pervasive Insights™ answers it, so the answers can be compared directly against whatever else is on your desk.

1. What human data was this calibrated against, and when?

A synthetic panel is only as good as the human research underneath it. Ask which studies were used, what they measured, and what year they were fielded, because a panel calibrated on a 2019 tracker will answer 2019 questions. Ask what happens when the data ages.

How Pervasive Insights™ answers this: Calibration runs against the client's own fielded studies, each carrying registry metadata: origin, year, market, sample size, and whether the study is current or retired. Every answer exposes which studies it drew on.

2. What is the reported error on this topic, and what is the error between two halves of a real study on the same topic?

A single accuracy number means nothing without the human benchmark beside it. Ask for the error on the specific topic you care about, not a site-wide average, and ask what the same measurement looks like when a real survey is compared against itself.

How Pervasive Insights™ answers this: Error is reported topic by topic on a report card. For scale: when a real survey is split randomly in half, the two halves agree within 5 points on about 85% of answers (58 consumer studies, 41,340 answer pairs). No synthetic method can beat a human study against itself, and Pervasive Insights™ reports both numbers on the same report card.

3. What happens on a topic you have not calibrated?

This is the question that separates a research instrument from a text generator. A model will always produce an answer; the question is whether the vendor lets it. Ask to see what the product returns for a topic with no underlying human data.

How Pervasive Insights™ answers this: A blank, not a guess. Topics that are not calibrated return no number and carry an explicit status, and results falling outside the human range are held for review rather than reported.

4. Is calibration fit being presented as prediction? Ask for a blind holdout.

Grading a panel against the same data it was built from measures construction, not accuracy, and it always looks good. The only honest test deletes real answer cells, predicts them blind, and reports the miss. Ask specifically whether the headline number is in-sample or held out.

How Pervasive Insights™ answers this: Validation runs transfer tests on human-answered questions excluded from calibration, and holdout certification where real answer cells are deleted and predicted blind, with the typical miss and the within-five-point rate reported per question family.

5. Do segment differences match the human gaps, or are they inflated?

Uncalibrated personas exaggerate the distance between customer segments, which can point a team at the wrong target entirely. Published benchmarks in 2026 put the inflation at two to four times. Ask for the segment-gap ratio: the synthetic gap divided by the human gap on the same measure.

How Pervasive Insights™ answers this: Each synthetic response is gated through the measured human distribution for its segment before generation, and fidelity is reported both overall and by key subgroup, because calibration can look strong overall while failing badly for one group.

6. If I run the same question twice, do I get the same answer?

Estimates that move every time you run them are not research. Ask whether generation settings are fixed, whether the vendor runs replicates, and whether a tolerance band is published alongside each number.

How Pervasive Insights™ answers this: Generation settings are fixed, replicates are run, and a tolerance band is published alongside every synthetic estimate, so you get both the number and how much it moves.

7. Does my data train a model that serves anyone else?

Your research is an asset, and pooling it into a shared model hands your advantage to your competitors. Ask directly whether your data trains anything, and whether you can be opted in by default.

How Pervasive Insights™ answers this: No. Calibration is per client and per study. Client data never trains a model, is never co-mingled with another client's data, and clients cannot be opted in.

8. Is every table labeled synthetic or human?

Synthetic and fielded numbers must never sit in a deck without a label saying which is which, because six months later nobody remembers. Ask to see a real deliverable and check the labeling.

How Pervasive Insights™ answers this: Panelists are labeled synthetic in every interface, and every answer carries a source label distinguishing retrieved human research, computed human data, and synthetic simulation. Synthetic data is never commingled with natural-person data in results.

9. What is it not for?

A vendor who cannot name a bad use for their own product has not tested it hard enough. Ask what they would refuse to sell you, and whether that refusal is enforced in the product or just in the sales conversation.

How Pervasive Insights™ answers this: It is not for high-stakes go/no-go decisions, regulated or legal claims, or novel topics with no calibration data. Question categories unsuited to simulation are refused by the product, and the standing recommendation is that high-stakes decisions are verified with human research.

Three things sold as the same product

The phrase “synthetic respondent” covers three very different things. The difference is what sits underneath the answer.