The demo is always impressive. Platforms respond quickly, outputs look clean, and the presenter knows which features to lead with. You leave the call without a clear reason to say no and without a clear reason to say yes.
That’s where most evaluations of AI market research offers stall. Not because the technology isn’t real, but because vendors build demos to show you what the platform can do under ideal conditions. Real research projects rarely run under ideal conditions.
The timing makes this harder, not easier. By 2025, 71% of organizations regularly use generative AI in at least one business function, with enterprise spending hitting $37 billion — a 3.2x increase in a single year, according to multiple industry sources. Trust hasn’t kept pace. A Harvard Business Review survey of 603 business and technology leaders found just 6% fully trust AI agents to handle core business processes — and 23% specifically flag anxiety about data output quality as a primary concern.
That gap — adoption accelerating, trust falling — has a cause. Many vendors now deploy AI in research operations without the research standards infrastructure to support it. The AI generates an output. Someone decides what to do with it. How that decision gets made, with what expertise, and with what accountability varies enormously across AI market research vendors.
What makes this hard to fix is that the feedback loop is broken too. When something goes wrong operationally, clients don’t always report it — they absorb the extra time, work around the problem, or quietly move on. When they do flag it, the gap between how a researcher describes what went wrong and how a developer interprets it is wide enough that the fix misses the point. The problem comes back in a slightly different form. Trust erodes not through a single visible failure but through a slow accumulation of breakages that nobody fully accounts for.
The five questions below cut through. They don’t cover every dimension of vendor evaluation, but they consistently surface the difference between platforms built for research and platforms built for research sales.
Ask What Has Gone Wrong
Most vendor conversations cover what went right. This one asks about what didn’t.
A technology provider with no honest answer to this either hasn’t deployed at scale or isn’t being straight with you. Complex routing logic in large questionnaires creates problems. Multi-language processing at the edges of a language set does too. So do teams using AI-generated output without adequate review. Vendors who have worked through these issues have changed something as a result. Ask them what — and listen for whether the change was structural or cosmetic.
Ask Which AI Model Powers the Product
A generic large language model applied to research tasks is not the same as a system built around the specific, rule-governed workflow of a quant survey. One is fast. The other understands what routing logic is, what a consent block needs to do, and what happens when a skip pattern conflicts with a question added late in the design process. Beyond capability, if a vendor uses a third-party AI system, you need to know what happens to your data inside it — who owns what the AI produces, and under whose terms.
Ask How Output Gets Validated Before It Reaches You
This is the question most AI market research vendors find hardest to answer with any specificity.
A real answer describes a structured validation process that runs before output reaches the client. Not a human who checks if it looks roughly right. A documented, repeatable process covering specific risk categories such as logic consistency, compliance, data quality, etc. If the vendor can’t describe that process in concrete terms, validation is either informal or non-existent. The output arrives. Someone makes a judgment call. That’s interpretation, and interpretation is where quality variation hides.
ResearchReady is how we handle this at CodexMR — a seven-area validation review that runs before any questionnaire moves to programming. We mention it here not as a pitch, but as a reference point for what a documented answer to this question looks like in practice.
Ask Where the Human Sits in the Workflow
“Human oversight” appears in every vendor pitch. Ask what it means in their actual process.
AI generates outputs. A human decides what to do with them. The quality of that decision depends on the expertise of the person making it and the structure they work within. A platform that routes AI-generated output through review by experienced quant researchers before anything reaches a client is a fundamentally different proposition from one that delivers AI output directly and calls that a workflow. The difference isn’t visible in a demo. Ask the vendor to walk you through the specific steps between AI output and client delivery — and who owns each one.
Ask Who Owns the Output
Buyers most often forget this question until it matters.
Some vendors integrate third-party AI technology whose terms and conditions include claims over the intellectual property of outputs their system produces. That can restrict how you use the research findings — particularly if the intended use is commercial. Ask before you sign. Get the answer in writing. No clear answer on output ownership is itself an answer.
Wrap Up
These five questions reflect what we’ve learned running quantitative research operations at scale — the gaps that consistently appear between what vendors promise and what research actually requires.
They’re also at the core of a formal industry standard. In March 2024, ESOMAR published a 20-question checklist for buyers of AI-based market research services. Since then, it has become the closest thing the industry has to a due diligence framework for AI in research — the questions serious buyers reach for when the conversation moves past the demo.
We’re publishing CodexMR’s public answers to all 20 shortly — a section-by-section response to every question on the list, including the ones that are genuinely difficult to answer well. In the meantime, the full checklist is worth reading before your next vendor conversation.
