The burgeoning landscape of AI-powered visibility reports, often presented with compelling slides and promises of market insight, is facing critical scrutiny. Matt Heinz, a prominent figure in B2B marketing and sales strategy, argues that the core data underpinning these reports is largely based on inference rather than direct observation. He contends that the rapid and unpredictable nature of AI-generated answers renders traditional "ranking" metrics largely irrelevant, and that the most valuable data—customer queries—is already in the hands of businesses themselves. Heinz advocates for a shift from relying on vendor dashboards to conducting internal corroboration audits of existing data sources.
The Illusion of Observed Data in AI Visibility Reports
Many vendors in the AI visibility space present similar, persuasive data points. However, a closer examination reveals that the source of this data is often opaque. Heinz points out that a significant portion of the information used to construct these reports is not derived from actual user queries submitted to AI platforms. Instead, it is frequently built upon inferred data.
For instance, some platforms acknowledge that AI search engines like ChatGPT and Perplexity do not publicly share query data, and a dedicated keyword tool for AI search does not exist. Their methodologies often involve combining data from Google Search Console, existing SEO keywords, and AI-generated prompts. While some companies, like Otterly, are transparent about their data sources and methodologies, others remain vague. One well-funded platform claims to utilize "real prompts submitted to AI platforms by actual users" without specifying how this data is obtained. Another platform generates suggested prompts based on a company’s website and industry context, then scores them against Google search trends—a process Heinz describes as "a proxy for a proxy."
This reliance on inferred data, rather than directly observed user queries, significantly impacts the confidence one can place in the insights derived from these reports. The distinction is crucial: inferred data is a projection, while observed data is a direct record of user behavior. The investment decisions made based on such reports could be fundamentally altered if this distinction is not clearly understood.
The Data Gatekeepers: Google, OpenAI, and Anthropic
The fundamental challenge in obtaining direct query data lies with the three major entities that possess it: Google, OpenAI, and Anthropic. These companies have little commercial incentive to share this granular information, which represents a significant competitive advantage and a rich source of user behavior insights.
Google recently introduced a new Search Console report for site owners focusing on AI Overviews and AI Mode. While this report provides data on impressions, pages, countries, devices, and dates, it conspicuously omits the actual queries and clicks associated with these AI-generated results. When questioned about the inclusion of query data, Google stated they are "continuing to work with website owners to understand what insights will be most helpful," a response that offers little immediate clarity on future data sharing.
Similarly, OpenAI’s advertising product provides advertisers with only aggregate performance data, such as views and clicks, but no access to individual chat histories or user memories. Anthropic, which develops Claude, has stated it will not run ads on its platform at all, eliminating a potential avenue for advertiser-sourced query data.
The absence of this critical query data in Google’s new reporting surface, a tool ostensibly designed for marketers, is a significant development. It suggests a deliberate decision rather than a mere oversight, and marketers should approach the current data landscape with caution.
The Ephemeral Nature of AI Answers and Rankings
Even if a comprehensive list of relevant prompts were available, the volatile nature of AI-generated answers presents another significant hurdle for traditional ranking metrics. A recent crowdsourced study involving nearly 3,000 prompt executions across ChatGPT, Claude, and Google’s AI Overview by 600 volunteers revealed a low probability of consistent results. The study found less than a one-in-a-hundred chance that two responses to the same prompt would return the same list of brands, and approximately one in a thousand chance of brands appearing in the same order.
Further research analyzing 815,000 prompt-page pairs indicated that after running the same prompt three times in ChatGPT, only about 2% of citations were consistently maintained. Another study tracking over 82,000 prompts across 17 weeks found that ChatGPT replaced roughly three-quarters of its cited sources week over week.
These findings underscore that "rank" in the traditional sense is largely meaningless in the current AI landscape. While visibility measured over hundreds of prompts and extended periods can offer directional insights, a snapshot of a company’s placement on a specific day is "noise and extremely temporary." The practice of reporting these fleeting rankings to executive boards, as many teams are reportedly doing, may be misrepresenting the true state of AI visibility.
Who is Really Being Read? The Dominance of Aggregators
A critical aspect often overlooked in AI visibility reports is the actual source of the information these AI systems use to generate answers. When AI models address questions within a specific industry or category, they are frequently not directly "reading" a company’s own website.
An analysis of over 100 million citations across ChatGPT, Google AI Mode, and Perplexity revealed that the most frequently cited domains were not company websites but rather aggregators and content platforms such as Reddit, Wikipedia, Medium, Forbes, LinkedIn, and YouTube. Notably, no vendor-specific sites cracked the top five across any of these platforms.
Additional research on Google’s Gemini found that the overlap between brands mentioned in an AI-generated answer and the domains actually cited can be as low as 30%. This means a company might be mentioned (a positive, but superficial outcome), while another platform provides the substantive evidence—a less desirable scenario for direct brand authority.
In this context, review sites continue to hold significant importance. G2, a prominent software review platform, observed that four out of five of its product pages were cited by AI systems more often than they were viewed by human users. While this data comes from a company selling vendor visibility and was tracked by an AI visibility software provider, it aligns with the broader trend: AI systems are leveraging established platforms for information.
The Buyer Enablement Imperative in the AI Era
The current state of AI visibility reporting highlights a critical shift towards buyer enablement. As AI tools increasingly inform early-stage buyer research, the traditional sales and marketing funnel is being redefined.
Research by 6sense, a company specializing in intent data, indicates that 94% of buying groups rank a preferred vendor before initial contact, and 77% proceed to purchase that preliminary favorite. The novelty lies in how this preliminary ranking is now partly assembled by AI, which sifts through forum threads and review pages on behalf of the buyer.
Gartner reports that 69% of B2B buyers prefer to validate AI-generated insights with a sales representative. This suggests that by the time a buyer engages with a sales team, their opinion is often already formed, and the conversation shifts from discovery to validation.
In response to these dynamics, Heinz proposes several strategic adjustments for businesses:
- Build Your Own Question Set: Instead of relying on external vendors, companies should leverage their own internal data. Sales calls, support tickets, win/loss debriefs, and community threads are invaluable, proprietary sources of buyer questions that cannot be replicated or sold by third parties.
- Optimize for Corroboration, Not Placement: In an environment where AI systems reward consistency across independent sources, the focus should be on ensuring that a company’s positioning, review profiles, customer stories, analyst coverage, and website all present a unified and truthful narrative. Inconsistencies, which may have previously led to a confused visitor, now result in a lost citation.
- Arm the Champion for Validation Conversations: In the AI-driven early stages of the buying journey, the "champion" within a buying group is often defending a pre-existing choice rather than discovering a new vendor. Enablement efforts should focus on providing champions with content tailored to different stakeholders (e.g., CFO, security lead) and proof points that facilitate consensus and accelerate group commitment to a decision.
- Ungate the Evaluation: If buyers can obtain answers about a product from an AI in seconds, requiring them to wait for demos or discovery calls is a disengaging tactic. Businesses should consider making evaluation processes more accessible, allowing buyers to get direct information without undue delays, lest they simply return to asking the AI.
The Corroboration Audit: A Deeper Dive into Data Consistency
The concept of a "corroboration audit" addresses the issue of internal data dissonance, which mirrors the external inconsistencies that AI systems can now identify. This audit process, akin to maintaining a living wiki for an Ideal Customer Profile (ICP) rather than a static document, focuses on aligning messaging and information across all external-facing touchpoints.
The audit itself is a structured process:
- Identify all external-facing content: This includes website copy, product descriptions, marketing collateral, sales enablement materials, review site profiles, social media content, and press releases.
- Map key messages and value propositions: Document the core claims made about the company’s offerings, target audience, and competitive advantages across each content source.
- Identify contradictions and inconsistencies: Compare the mapped messages to pinpoint areas where different versions of the truth are being presented. This can include discrepancies in target market definitions, feature sets, pricing models, or core value propositions.
- Prioritize and rectify discrepancies: Develop a plan to address identified inconsistencies, aiming for a unified and accurate representation of the company across all platforms.
The outcome of such an audit is a list of contradictions, many of which may predate the rise of AI but are now more readily exposed. The AI, acting as a meticulous reader, can surface these inconsistencies at scale, impacting a company’s perceived reliability.
The Double-Edged Sword of AI Pattern Matching
The pattern-matching capabilities of AI systems, while powerful, also present a cautionary note. These systems draw from overlapping pools of data for both buying and employment-related questions. An anecdote about a CEO attempting to manipulate Glassdoor reviews by paying for positive ones highlights a past strategy that may no longer be effective.
As AI models become more sophisticated, clusters of glowing reviews that contradict other available signals are likely to be perceived as "noise" by systems designed to reward agreement. This could lead to AI summaries that highlight positive product attributes while simultaneously pointing to negative employee treatment, prompting buyers to draw their own conclusions about a company’s overall viability.
Marketers, who often seek control over their brand narrative, are facing a diminishing degree of it as AI takes a more prominent role in information synthesis. The key takeaway is that while external control may be limited, internal control over the consistency and truthfulness of all information sources remains paramount. In this new paradigm, the job of building buyer confidence evolves, with AI acting as an impartial, albeit unvisitable, grader of a company’s consistent narrative. This presents new challenges, constraints, and ultimately, opportunities for businesses to adapt and thrive.
