Intro
The AI visibility category is young and noisy. New tools launch every quarter. Existing platforms rebrand overnight. Agencies bolt “GEO” onto their service decks. The word “methodology” appears everywhere, and means very little.
For a CMO, head of digital, or head of distribution, the hard part is no longer “should we measure AI visibility?”. It is “how do we tell a serious vendor from a repackaged one?”.
This page offers a framework. Ten questions to ask any AI visibility vendor, three failure modes to recognise, and the red flags that signal a tool will not survive a quarterly retest.
Why category-wide buyer guidance is needed
AI visibility looks like SEO from the outside. It is not. The signal is shorter-lived, the engines are opaque, and the answer surface changes weekly. A tool that produced a credible report in January can produce a misleading one in May without anyone noticing.
A premium brand carries reputational risk that a startup does not. A wrong claim about a heritage hotel, a luxury maison, or a regulated service product can be repeated by AI engines for months. That makes the choice of measurement tool a brand-safety decision, not just an analytics decision.
A buyer’s framework has to test three things: whether the vendor has a real methodology, whether the data layer is auditable, and whether the work is repeatable across quarters and engines.
Three failure modes in the current category
These three patterns appear in most sales conversations. Each looks credible at the demo stage and breaks down at the second retest.
Failure mode 1 — The repackaged SEO platform. Established SEO tools have added an “AI” tab. The tab usually shows keyword rankings filtered through an AI-shaped chart, or a handful of generic prompts run once a month. The underlying methodology is still organic search. AI engines are not search engines, and a tool that treats them as such will miss the citation layer, the answer-position layer, and the commercial-risk layer.
Failure mode 2 — The prompt-logger-only tool. A second category captures AI answers at scale and exposes a dashboard. The product is real, but the methodology is thin: no co-designed prompt set, no competitor frame, no QA loop, no scoring. The buyer gets thousands of rows and no decision. Volume is not measurement.
Failure mode 3 — The one-off agency audit. A consultancy delivers a 40-page PDF, walks the brand through it, and leaves. Three months later the answers have moved and nothing in the deck is still true. The agency is selling artefacts, not a measurement system. A static audit cannot track a moving target.
Ten questions to ask any AI visibility vendor
Ask these ten questions in order. The combined answers separate methodology from marketing.
1. What is your scoring methodology, and how is it documented?
What good looks like: a written framework with named dimensions, defined inputs, and a version number. If the methodology cannot be shared on paper, it does not exist. See the Capston Core methodology for the structure a serious framework should have.
2. Can I see the underlying evidence for any score?
What good looks like: every score traces back to dated, captured AI answers with model and version metadata. A score without an evidence layer is an opinion.
3. How is the prompt set built, and how is it locked?
What good looks like: prompts are co-designed with the brand, organised by intent (discovery, comparison, trust, conversion), and frozen for the measurement period. Ad-hoc prompts mean unrepeatable results.
4. Which AI engines do you cover, and how do you decide?
What good looks like: engine coverage is matched to the brand’s actual markets and audiences, not a default vendor list. The vendor should be able to defend each engine choice.
5. How do you handle model updates and answer drift?
What good looks like: model versions are logged, retests are scheduled, and answer drift is reported as a category — not hidden inside aggregate scores.
6. What QA standards apply to the captured data?
What good looks like: a documented QA process covering capture integrity, prompt fidelity, language consistency, and reviewer sign-off. Quality assurance is a process, not a promise.
7. Is the measurement repeatable quarter on quarter?
What good looks like: the same prompt set, the same competitor set, the same engine set, run on a fixed cadence. Without repeatability there is no trendline, only snapshots.
8. How does the tool integrate with our existing stack?
What good looks like: clean exports, an API or scheduled feed, and a defined hand-off to the brand’s analytics and CRM. A tool that lives only inside its own dashboard adds work, not signal.
9. What is the partnership model after the first report?
What good looks like: a stated cadence, a named team, and a clear escalation path. The first report is the easy part; the second, third, and fourth are where most vendors lose discipline.
10. What ROI horizon do you commit to, and how is it measured?
What good looks like: a defined timeframe for visibility gains, tied to specific dimensions (citation share, answer position, fact accuracy), with a stated method for attribution. ROI claims without a horizon are sales copy.
Red flags to watch for
A vendor evaluation can move quickly once the red flags are named.
- The word “AI” appears more often than the word “prompt” or “evidence” in the sales deck.
- The product demo uses a single brand example and refuses to run a live prompt against the buyer’s own brand.
- The score is a single number with no dimensions behind it.
- The vendor cannot show a prior client’s quarter-on-quarter retest.
- The pricing model rewards new audits over ongoing measurement.
- The methodology page on the vendor’s site is shorter than the case studies page.
- “Coverage” is described as “all major AI engines” with no list.
- The QA section is one paragraph.
None of these on its own is fatal. Three or more together is a pattern.
How this fits into Capston Core
Capston Core is built around the answers this framework asks for: a documented methodology, an auditable evidence layer, locked prompt sets, defined engine coverage, QA standards, and a repeatable quarterly cadence. The Capston methodology, the data evidence layer, the QA standards, and the certification framework exist precisely so a premium brand can answer these ten questions without ambiguity.
Buyers who run this framework with Capston Core get the same documentation we would expect from any rigorous vendor. That is the point of publishing the framework.
→ Back to Capston Core
FAQ
Is one AI visibility tool enough, or do we need several?
For most premium brands, one rigorous measurement system is enough, provided it covers the relevant engines and is run on a fixed cadence. Multiple tools without a shared methodology produce conflicting scores.
Should we run an internal audit before talking to vendors?
A short internal review of current AI answers on five to ten branded prompts is useful before any sales call. It anchors the vendor conversation in observed reality rather than a generic demo.
How long does a proper vendor evaluation take?
Two to four weeks is realistic: one week to brief, one to two weeks for vendors to respond with methodology documentation and a live prompt sample, and a final week to compare evidence layers side by side.
What if our existing SEO agency offers an AI visibility module?
Apply the same ten questions. If the methodology, evidence layer, and QA standards are documented and defensible, the existing relationship is an asset. If the module is a repackaged keyword report, it is not.
Final CTA block
Run this framework with a vendor that publishes its answers.
Talk to Capston Core
Read the methodology