Asro
Marketers have grappled with the complex question of attribution for generations, long before the advent of sophisticated digital tracking tools. The challenge of accurately assigning credit for conversions remains a persistent hurdle, and the realm of Paid Search is no exception. While various attribution models have emerged over the years, none have definitively solved the puzzle. This is where incrementality testing emerges as a powerful, data-driven solution, promising to move beyond the limitations of traditional methods and provide a clearer understanding of marketing’s true impact.
At its core, incrementality testing addresses a fundamental question: "How much of this would have happened anyway?" This concept, known as counterfactuality, is critical because traditional attribution models are often inherently biased. A prime example is relying solely on platform-specific analytics, such as Google Analytics evaluating Google Ads campaigns. Such systems can inadvertently overstate the performance of the platform they are integrated with, leading to an inaccurate picture of marketing effectiveness. Furthermore, a multitude of platforms often vie for credit for the same leads and conversions, creating a crowded landscape where every channel claims victory. Even within Paid Search, branded search campaigns frequently dominate, showcasing impressive key performance indicators (KPIs) without adequate acknowledgment of the upstream tactics that fostered the demand and ultimately drove those conversions. Incrementality testing aims to dismantle these biases by creating controlled, real-world testing environments designed to attribute credit where it is genuinely earned.
Understanding Incrementality Testing: A Shift from Correlation to Causation
Incrementality testing is fundamentally about measuring whether specific marketing actions genuinely influence outcomes. It operates on the principle of experimentation: establishing a baseline of standard performance and then manipulating a specific variable within the marketing mix to observe its direct effect on overall results. Unlike attribution modeling, which often analyzes past performance to assign credit, incrementality testing is a proactive and deliberate process of changing inputs to measure the resulting outputs. This methodology allows marketers to move from observing correlations to establishing causation.
Paid Search programs, in particular, stand to benefit immensely from a deeper understanding of tactical incrementality. While branded search campaigns are often highlighted for their strong KPIs, the origin of those purchases, leads, and other conversions can be opaque. The critical question becomes: What role do top-of-funnel strategies, such as Non-Brand Search and demand generation efforts, play in nurturing potential customers and ultimately influencing their decision to convert via branded search terms? Savvy Paid Search managers are increasingly leveraging incrementality testing to precisely quantify the contributions of each component within their broader program.
Designing Effective Incrementality Tests for Paid Search
While the concept of setting up an incrementality test might initially seem daunting, establishing an ongoing testing structure is crucial for optimizing Paid Search programs, eliminating wasteful spending, and fostering long-term success. Several key factors must be considered when developing a robust testing framework.
Defining Success: The Cornerstone of Any Test
The initial and most critical step in any incrementality test is to clearly define what the advertiser aims to evaluate. This involves precisely identifying the variable under observation, whether it’s the true incremental contribution of Non-Brand Search campaigns or the often-unattributed value generated by video advertising. When determining what to measure, it is imperative to not only identify relevant KPIs but also to establish a definitive "source of truth" that all stakeholders can agree upon. For an e-commerce company, a well-defined objective might be: "We are measuring the impact of Non-Brand Search by monitoring for an overall lift in Sales and Revenue within Shopify."
Pro-Tip: To effectively measure any form of lift, an advertiser must possess a clear understanding of baseline performance. This baseline serves as the benchmark against which performance is compared, allowing for an estimation of what results would have been without the influence of the tested variable. The difference between this estimated baseline and the actual observed performance is then attributed as the incremental lift.
Defining Test Parameters: Methodologies for Measurement
Once the objective is clearly defined, the next crucial step involves selecting the appropriate methodology to structure the test for maximum clarity. Geo holdout and lift tests are two of the most widely adopted testing structures among experienced Paid Search managers.
Geo Holdout: Isolating Impact Through Geographic Control
Geo holdout tests offer a relatively straightforward approach to incrementality testing for Paid Search managers. In its simplest form, specific campaigns or campaign types are temporarily paused in designated geographic regions for a defined period. Following this period, the performance data from the test group (where campaigns were paused) is meticulously compared against a control group (where campaigns continued as normal). A significant logistical challenge with this method can be ensuring that the chosen source of truth provides granular, state-level conversion data, which is essential for accurate analysis. The effectiveness of this method hinges on the ability to create a statistically significant difference between the test and control regions.
Lift Tests: User-Centric Measurement of Impact
Often referred to as user holdout tests, lift tests involve exposing one group of users to a specific set of advertisements while simultaneously comparing their results against a separate group of users who are not served those ads. Because this methodology operates at the user level, lift tests typically focus on platform-specific metrics. Common examples include Google Ads’ Brand Lift and Conversion Lift studies, frequently observed within video-based campaign strategies. These tests are designed to isolate the impact of a particular ad exposure on user behavior.
Optimizing Timing: Mitigating External Influences
External factors can significantly influence the outcome of an incrementality test, making it imperative to design experiments that minimize these potential disruptions. Variables such as ongoing sales promotions, seasonality, anticipated shifts in competitive landscapes, and even internal operational changes can all sway performance during a given period. Using the e-commerce example, potential confounding factors could include shipping delays or products unexpectedly going out of stock. It is not only the duration of the testing period itself that is critical, but also the strategic planning around it.
Pro-Tip: A comprehensive incrementality test should incorporate a "halo period" after the primary test concludes. During this phase, overall performance is monitored to ascertain if there is a sustained, significant change within the test group as they transition back to standard marketing activities. This helps to capture any lingering effects of the tested variable.
Beyond calendar timing, a major consideration for any incrementality test is determining its optimal duration. While budget constraints often play a significant role, stakeholders must also reach a consensus on the "minimum detectable effect" (MDE). The MDE represents the smallest performance fluctuation that is sufficient to convince all parties that the test results are conclusive. In simpler terms, it’s the minimum observed impact required to trust the test findings. It’s important to note that MDE is distinct from statistical significance; MDE is estimated before a test is launched, whereas statistical significance is determined after the test concludes, based on the collected data.
The MDE is also inherently linked to the sample size being observed. A larger affected population generally leads to greater confidence in the test results. Therefore, it is crucial for advertisers to estimate the potential audience size and agree upon what constitutes a meaningful change in performance that is likely attributable to the test, rather than mere period-to-period fluctuations inherent in business operations.
Communicating and Reporting Results: Building Trust and Driving Action
Effective reporting is arguably the most crucial element of any incrementality test, particularly when seeking buy-in and maintaining stakeholder trust. When presenting the recommended nature and structure of a test, objections are not uncommon. For instance, presenting a geo holdout test might elicit concerns such as, "Turning off ads in ten states sounds like losing money on purpose!"
This is precisely why advertisers must articulate any anticipated risks with utmost clarity, reiterate the MDE required to achieve statistical confidence, and provide an estimated timeline for performance normalization once the test concludes. Furthermore, as experienced Paid Search managers well know, a consistent reporting cadence is essential for ongoing monitoring of the test’s progress. These elements are vital for securing approval from broader teams and ensuring the test proceeds smoothly.
Interpreting and Reporting Findings: A Data-Driven Narrative
Given that the KPIs and the source of truth were agreed upon prior to the test’s launch, there should be minimal surprises when it comes time to review the results. To conduct this review effectively, advertisers should restate the anticipated MDE and clearly identify the actual change driven during the testing period.
For example, if the hypothetical e-commerce company proceeds with a geo holdout test by pausing Non-Brand Search campaigns in ten states, the results report should detail the performance in both the test group and the control group.
To effectively communicate this narrative, the results must encompass the following critical data points:
- Baseline Performance: A clear representation of performance metrics prior to the test’s commencement.
- Test Period Performance: Data from both the test group (where the variable was manipulated) and the control group (where standard operations continued).
- Observed Lift/Decline: The quantifiable difference in performance between the test and control groups during the test period.
- Incremental Value Calculation: A financial or quantitative estimation of the true impact of the tested variable.
- Post-Test Performance Analysis: Data reflecting performance after the test concludes and normal marketing activities resume, including the halo period if implemented.
By observing such a comprehensive series of data points, advertisers can present a well-rounded summary of the test’s impact. If the paused campaigns were indeed driving business objectives, their suspension in the test locations should result in a discernible decrease in sales compared to the pre-test period. Crucially, if this decline is directly attributable to the pausing of Non-Brand Search, the non-test locations are unlikely to exhibit a similar downturn.
The performance disparity between the test and control locations serves as a strong indicator that Non-Brand Search is delivering genuine incremental value to the bottom line, even if these campaigns haven’t historically received full attribution for their contributions. Moreover, if sales rebound in the test locations shortly after the test concludes, this provides further corroborating evidence that Non-Brand Search was indeed responsible for driving those sales.
Navigating Inconclusive or Negative Results: Opportunities for Optimization
There will inevitably be instances where incrementality tests reveal that certain campaigns are ineffective or fail to drive a statistically significant result. These situations present a prime opportunity for strong advertisers to demonstrate their partnership capabilities. When unfavorable results emerge, it is imperative to be transparent about them, explain them clearly, and propose a concrete action plan.
What if pausing Non-Brand Search campaigns in the hypothetical example did not yield a significant impact on overall performance? The data should articulate this concretely, and an advertiser should feel empowered to explore the underlying reasons. Perhaps the campaigns are targeting the wrong keyword themes, or the advertiser is already saturating the market through other channels like Performance Max, Meta, or even emerging AI-driven advertising tools, causing Non-Brand Search efforts to be drowned out. Regardless of the cause, investigating the "why" opens the door for further testing and optimization.
Action plans are often built upon initial findings that are not immediately successful. This underscores the value of developing an ongoing testing framework rather than treating each test as an isolated, one-off event. This iterative approach fosters continuous learning and refinement.
Incrementality Testing as a Foundational Practice
Incrementality testing is not a singular exercise aimed at finding a definitive answer based on a snapshot in time. Instead, it should be embraced as an ongoing practice that continuously enhances the effectiveness of marketing accounts. In today’s complex omnichannel environment, no single system perfectly measures the effectiveness of every channel. However, a robust incrementality testing framework cultivates confidence, ensures strategies remain current and relevant, and, most importantly, directly links Paid Search efforts to overarching business objectives. It can readily serve as the cornerstone of an effective account management framework. Businesses do not typically engage a Paid Search manager or agency solely for improved click-through rates; they do so with the anticipation that the partnership will ultimately drive tangible business growth. Incrementality testing is the indispensable tool that validates and amplifies this fundamental assumption, transforming aspiration into demonstrable reality.
Sagoh
Nana Muazin
