ESOMAR’s 20 questions for buyers of AI-based research, answered

Pervasive Insights™ is a synthetic research panel built on a client's own human research: a searchable library of every study the client has run, and synthetic respondents calibrated to how that client's real customers answered, with the error reported topic by topic. It is a City Research Solutions product. City Research has run consumer research since 1979.

These are ESOMAR’s 20 questions for buyers of AI-based services for market research and insights, answered for Pervasive Insights™. Question wording is reproduced from ESOMAR, “20 Questions to Help Buyers of AI-Based Services for Market Research and Insights,” © 2024 ESOMAR, issued March 2024, with attribution per ESOMAR’s terms. The answers are the responsibility of City Research Solutions.

1. What experience and know-how does your company have in providing AI-based solutions for research?

Pervasive Insights™ is built and operated by City Research Solutions, a US market research consultancy in its third decade of practice, with a quarter million measured human respondents across more than a thousand studies. The platform was built by researchers to extend primary research: the AI engineering exists in service of research methodology, not the other way around. The calibration approach at its core was developed, tested, and certified in-house against our own clients' primary data.

2. Where do you think AI-based services can have a positive impact for research? What features and benefits does AI bring, and what problems does it address?

Three problems, specifically. First, buried findings: most organizations re-purchase answers their own archives already contain, and AI-powered retrieval over a curated report library ends that. Second, wasted data: reports historically present a fraction of what studies collect, and AI-assisted analysis over harmonized respondent-level datasets lets clients compute the cuts and crosses nobody ran. Third, the cost of exploration: calibrated synthetic respondents let teams screen concepts, language, and pricing structures in hours, reserving human fieldwork for the decisions that deserve it. The common thread is that AI multiplies the value of real research rather than replacing it.

3. What practical problems and issues have you encountered in the use and deployment of AI? What has worked well and how, and what has worked less well and why?

We will answer with our scars. Raw language models failed as respondents in three measurable ways: they anchored on famous brands regardless of measured preference, inflated purchase intent severalfold, and homogenized segment voices. Prompt engineering alone did not fix the quantitative failures; enriching personas with real customer language improved qualitative authenticity but still not distributions. What worked was architectural: a two-pass design where measured human data decides response distributions before the model generates anything. Ongoing lessons include segments with thin underlying data running hot or cold until their cards are rebuilt from richer data, and coarse rating scales producing rounding artifacts at fine tolerances. We document misses in client-visible report cards rather than patching silently.

4. Can you explain the role of AI in your service offer in simple, non-technical terms in a way that can be easily understood by researchers and stakeholders? What are the key functionalities?

Yes, in one paragraph. The portal does three things: finds and cites your existing research reports; computes fresh analysis from your raw survey data; and simulates surveys, focus groups, and interviews with synthetic customers built from your data. The AI model supplies language, summarizing, conversing, giving synthetic respondents their voices. It never supplies the numbers: quantitative results come from your measured data directly, or from simulations whose distributions are constrained by your measured data. Model as voice, data as substance.

5. What is the AI model used? Are your company's AI solutions primarily developed internally or do they integrate an existing AI system and/or involve a third party and if so, which?

The language layer integrates commercial frontier models (currently Anthropic's Claude family) under enterprise API terms. Everything that makes the service a research instrument, the calibration architecture, the probability gating, the question-mapping constellation, the harmonized data layer, the evidence and confidence machinery, is developed and owned internally by City Research Solutions. Third-party model providers receive prompts under enterprise terms that exclude training on our traffic; they never receive client datasets wholesale.

6. How do the algorithms deployed deliver the desired results? Can you summarise the underlying data and the way in which it interacts with the model to train your AI service?

No client data trains any model, ours or a third party's, and clients cannot be opted in. Client data interacts with the model at answer time only: retrieval supplies relevant report passages for citation; analysis runs conventionally (statistically) on raw datasets with the model narrating results; and for synthetic respondents, measured human distributions gate each panelist's quantitative disposition before the model generates the response voice. The underlying data is the client's own primary research, curated into a registry with provenance and dates. Primary language of the corpus is US English; other markets' studies are scoped to answer only questions about those markets.

7. What are the processes to verify and validate the output for accuracy, and are they documented? How do you measure and assess validity? Is there a process to identify and handle cases where the system yields unreliable, skewed or biased results? Do you use any specific techniques to fine-tune the output? How do you ensure that the results generated are 'fit for purpose'?

Documented and client-visible. Synthetic validation runs three tiers against the client's own primary research: benchmark replication (distribution-level comparison per segment, reported in percentage points), transfer tests on human-answered questions excluded from calibration, and holdout certification (real answer cells deleted and predicted blind, typical miss and within-five-point rates reported per question family). Results publish to a standing calibration report card in the client's portal. We report two different numbers and label them. Calibration fit, which is how well the panel matches the studies it was tuned on, runs over 98% of calibrated cells within 5 points. Blind holdout, which is how well the panel predicts human answers it never saw, is the number that counts: on the certified floor tier the median miss is about 6.8 points and 42% of cells land within 5. For scale, when a real survey is split randomly in half the two halves agree within 5 points on about 85% of answers, with a median gap of 1.9 points (58 consumer studies, 41,340 answer pairs). Fit is never presented as prediction, and unsourced claims are refused. On hallucination: generative output is grounded in retrieved, cited sources; claims that cannot be sourced are refused rather than generated; out-of-scope questions receive explicit refusals. Data recency is metadata on every answer (study years, most-recent-leads, retired-study labeling). Synthetic data is always flagged as synthetic and never commingled with natural-person data in results; every result labels its source type.

8. What are the limitations of your AI models and how do you mitigate them?

Synthetic respondents cannot recall real events, cannot represent populations the client's data never measured, run wider on highly novel stimuli, and inherit the currency of the latest human wave. Mitigations: a four-level evidence hierarchy (exact calibrated questions, near neighbors, client-level response mapping, and a clearly labeled universal layer), visible confidence tiers on every synthetic result, refusal of question categories unsuited to simulation, and a standing recommendation, built into our own guidance, that high-stakes decisions be verified with human research. Limitations are documented in the client's portal, not in a footnote of a contract.

9. What considerations, if any, have you taken into account, to design your service with a duty of care to humans in mind?

Two populations. For research participants: no personally identifying information from any human respondent enters a synthetic panelist, a prompt, or an answer; panelists are statistical composites with invented identities. For decision-makers and the people their decisions affect: the service is engineered against confident misinformation, sources labeled, confidence tiers shown, unsupported claims refused, because the real duty-of-care risk in research AI is a wrong number believed. We would rather return “we cannot answer this” than a fluent guess.

10. Transparency: How do you ensure that it is clear when AI technologies are being used in any part of the service?

Structurally clear: the product is named a synthetic panel, panelists are labeled synthetic in every interface, every answer carries source labels distinguishing retrieved human research, computed human data, and synthetic simulation, and AI-generated deliverables are identified as portal outputs. No user can mistake a synthetic result for fielded human data without ignoring the labels on it.

11. Do you have ethical principles explicitly defined for your AI-driven solution, and how in practice does that help to determine the AI's behaviour? How do you ensure that human-defined ethical principles are the governing force behind AI-driven solutions?

Yes, and they are enforced in code, not posters. Standing rules with technical enforcement include: never present retired or superseded research as current (quarantine and labeling); never invent findings absent from sources (grounded generation with refusal); never let synthetic results outrank existing human answers; always disclose confidence; fail closed when client context is unresolved. Human validation gates every calibration round and every material change; outputs feeding decisions carry an audit trail from claim to source in two clicks.

12. Responsible Innovation: How does your AI solution integrate human oversight to ensure ethical compliance?

Human-in-the-loop at every consequential layer: researchers design and review calibration rounds; changes to a client's panel ship as versioned, reviewed data rounds rather than silent drift; professional human testers exercise portals before client teams receive them; and the curated study registry and question-map constellation function as human-engineered knowledge structures that constrain what the model may assert, with all machine-generated additions reviewed by researchers before use.

13. Data quality: How do you assess if the training data used for AI models is accurate, complete, and relevant to the research objectives in the interests of reliable results and as required by some data privacy laws?

By dataset truth, not document claims. Calibration inputs are verified at the respondent-data level: base sizes confirmed from dataset row counts rather than report summaries, variables mapped and reconciled during harmonization, study provenance and status curated in a registry, and screener-only respondents distinguished from completed interviews. Representativeness is handled honestly: calibration tolerances widen where underlying human bases are small, and that widening is disclosed on the report card rather than averaged away.

14. Data lineage: Do you document the origin and processing of training or input data, and are these sources made available?

Yes, end to end. Every study in a client corpus carries registry metadata: origin, year, market, sample size, status (current or retired). Every portal answer exposes its lineage: which documents were retrieved, which datasets were computed on, which calibration anchors applied. The client can walk any claim back to its source study inside the interface.

18. Data ownership: Do you clearly define and communicate the ownership of data, including intellectual property rights and usage permissions?

Yes, contractually per engagement: the client's research data remains the client's property; City Research Solutions holds it solely to service the engagement; it is never published, pooled, or uploaded into public AI tools. The calibration methodology, platform, and the anonymized universal response layer remain City Research Solutions intellectual property.

20. Ownership: Are you clear about who owns the output?

Yes: deliverables produced through the client's portal, answers, memos, decks, exports, belong to the client under the engagement agreement, with no third-party model provider holding IP claims over outputs under our enterprise terms. City Research Solutions retains ownership of the underlying platform and methodology, never of the client's findings.

Answers to questions 15, 16, 17 and 19 are being finalized.