Dwi Wanna
Marketers have grappled with the complexities of marketing attribution for generations, a challenge that predates the advent of digital tracking and has persisted through the evolution of paid search campaigns. While paid search offers powerful tools for reaching target audiences, it has not inherently solved the attribution dilemma. Enter incrementality testing, a methodology that promises to deliver a more accurate understanding of marketing impact by focusing on a fundamental question: how much of a given outcome would have occurred anyway? This concept, known as counterfactuality, is crucial because traditional attribution models are often susceptible to platform biases, such as relying on Google Analytics to evaluate the efficacy of Google Ads campaigns. Such self-referential assessments can lead to an overestimation of a platform’s direct impact, while other channels may not receive appropriate credit.
The landscape of digital advertising is characterized by a constant tug-of-war for credit. Each platform, from search engines to social media, often claims ownership of leads and conversions. Even within the realm of paid search, branded search campaigns frequently capture the spotlight due to their strong Key Performance Indicators (KPIs). However, this often overlooks the crucial top-of-funnel activities that nurture potential customers and ultimately drive them to search for branded terms. Incrementality testing aims to dismantle these biases by creating controlled, real-world experimental environments designed to assign credit accurately, reflecting the true contribution of each marketing touchpoint.
Incrementality testing represents a paradigm shift from retrospective analysis of past campaigns to a proactive approach of measuring the impact of specific marketing actions. It involves establishing a baseline of standard performance and then deliberately altering a variable within the marketing mix to observe its effect on overall outcomes. Instead of merely evaluating historical data to attribute results, incrementality testing is a direct intervention, changing inputs to precisely measure what happens as a result.
Paid search programs, in particular, stand to benefit significantly from a deeper understanding of tactical incrementality. While branded search campaigns often exhibit impressive metrics, their success is frequently a consequence of earlier marketing efforts. Non-brand search and demand generation tactics, for instance, play a vital role in filling the marketing funnel, informing, and persuading users who may eventually convert through branded search terms. Forward-thinking paid search managers are increasingly adopting incrementality testing to precisely quantify the contributions of each component of their campaigns.
Evaluating the Incrementality of Paid Search Tactics
Implementing an incrementality test might initially appear daunting, especially for those without a statistical background. However, establishing an ongoing testing structure is instrumental in eliminating wasteful expenditure and fostering long-term success in paid search programs. Several critical factors must be considered when developing such a framework.
Defining Success: The Cornerstone of Any Test
The initial step in any incrementality test is to clearly define the objective. What specific aspect of marketing performance is the advertiser seeking to evaluate? This could range from understanding the true contribution of non-brand search campaigns to quantifying the unattributed value of video advertising. When determining what to measure, it is paramount to identify not only relevant KPIs but also to establish a mutually agreed-upon source of truth that all stakeholders can trust. For an e-commerce business, this might translate to a clear objective: “We are measuring the impact of Non-Brand Search by monitoring for an overall lift in Sales and Revenue within Shopify.”
To effectively measure any form of lift, an advertiser must possess a reasonable understanding of baseline performance. This baseline serves as an estimation of what performance would look like without the influence of the variable being tested. The difference between this estimated baseline and the actual observed performance is then attributed to the tested element as incremental lift. Without this historical context, discerning genuine impact from natural fluctuations becomes impossible.
Defining Test Parameters: Structuring for Clarity
Once the measurement objective is defined, the next crucial step involves selecting the methodology for structuring the test to yield the clearest possible insights. Among paid search managers, Geo Holdout and Lift Tests are widely recognized as two of the most popular and effective testing structures.
Geo Holdout Tests: A Geographic Approach to Measurement
Geo holdout tests offer a relatively straightforward incrementality testing option. The fundamental principle involves pausing specific campaigns or campaign types in designated geographic regions for a predetermined period. Following this pause, the performance data from the test group is meticulously compared against that of a control group. A significant challenge with this method can be the ability to extract state-level conversion data from the chosen source of truth, requiring careful consideration of data infrastructure.
Lift Tests: User-Centric Measurement
Often referred to as user holdout tests, Lift tests operate by exposing one group of users to a set of advertisements while withholding these ads from a separate, comparable group. This user-level approach typically relies on platform-specific metrics. Common examples include Google Ads’ Brand Lift and Conversion Lift studies, frequently employed in video-based campaign evaluations. These tests are designed to isolate the impact of ad exposure at an individual user level.
Optimizing Timing: Mitigating External Influences
External factors can significantly influence the outcome of an incrementality test. Therefore, it is imperative to design the test in a manner that mitigates these potential distortions. Elements such as ongoing sales or promotions, seasonal trends, anticipated shifts in competitive landscapes, and even internal operational changes can all sway performance within a given period. For an e-commerce business, potential disruptors could include shipping delays or stockouts of popular products. It is not solely the testing period itself that requires careful consideration; the lead-up and aftermath are equally important.
A comprehensive incrementality test should incorporate a built-in "halo period" after the test concludes. During this phase, overall performance is monitored to ascertain if there is a significant, lasting change within the test group after resuming normal marketing activities. This helps to understand the sustained impact, if any, of the tested intervention.
Beyond calendar timing, a critical consideration for any incrementality test is its duration. While budget constraints often play a role, stakeholders must also agree on a Minimum Detectable Effect (MDE). The MDE represents the smallest fluctuation in performance deemed sufficient to convince all parties that the test results are conclusive. In simpler terms, it answers the question: how much of an impact must be observed to confidently declare the test results reliable? It is crucial to differentiate MDE from statistical significance. MDE is an estimate established before a test commences, whereas statistical significance is determined after the test concludes, based on the collected data.
The MDE itself will vary depending on the sample size under observation. A larger affected population generally leads to greater confidence in the test results. Consequently, advertisers must accurately estimate audience size and agree on what constitutes a genuine change in performance that is demonstrably a result of the test, rather than mere period-to-period business fluctuations.
Clarifying Success Details: Securing Buy-In and Maintaining Trust
Reporting stands as perhaps the most critical element of any incrementality test, particularly in securing stakeholder buy-in and fostering enduring trust. When proposing the recommended nature and structure of a test, it is not uncommon for stakeholders to express initial objections. For instance, any advertiser who has pitched a geo holdout test has likely encountered the sentiment, “Turning off ads in ten states sounds like losing money on purpose!”
This is precisely why advertisers must articulate anticipated risks with utmost clarity, reiterate the MDE required for statistical confidence, and provide an estimate of the time it might take for performance to normalize post-test. As any seasoned paid search manager understands, a regular reporting cadence is essential for continuous monitoring of the test. These transparent communication strategies are vital for obtaining approval from broader teams and ensuring alignment.
Interpreting and Reporting Results: Communicating the Impact
Given that the KPI and source of truth were agreed upon prior to test launch, the review of test results should ideally reveal few surprises. To conduct this review effectively, advertisers should restate the anticipated MDE and clearly identify the actual change in performance observed during the testing period.
For example, if the hypothetical e-commerce company pauses non-brand search campaigns in ten states for a geo holdout test, the results report should detail what transpired in that test group alongside the control group.
To effectively convey this narrative, results must incorporate several key data points:
- Baseline Performance: A clear depiction of performance metrics before the test commenced.
- Test Group Performance: Metrics from the geographic regions or user groups where the marketing variable was altered.
- Control Group Performance: Metrics from the geographic regions or user groups that remained unaffected.
- Observed Lift/Decline: The quantitative difference in performance between the test and control groups.
- Incremental Contribution: The calculated value attributed directly to the tested marketing activity.
- Cost of Test vs. Incremental Value: A comparison of the investment in the test against the incremental revenue or profit generated.
- Post-Test Performance: Data from the halo period to assess any lasting impact.
Observing this comprehensive suite of data points allows the advertiser to present a well-rounded summary of the test’s impact. If the paused campaigns genuinely drive business objectives, their discontinuation in test locations should lead to an observable decrease in sales during the test period compared to the pre-test period. If this decline is directly attributable to pausing non-brand search, the non-test locations are unlikely to experience a similar drop. The differential performance between the two location sets serves as a robust indicator that non-brand search is driving genuine incremental value to the bottom line, even if these campaigns do not always receive full attribution credit. Should sales rebound in the test locations shortly after the test concludes, this further validates the role of non-brand search in driving e-commerce sales.
Addressing Inconclusive or Negative Results: A Pathway to Optimization
There will inevitably be instances where incrementality tests reveal that certain campaigns are ineffective or are not driving a significant enough result to move the needle. This is an opportunity for strong advertisers to demonstrate their integrity as partners. When unfavorable results emerge, honesty and clarity are paramount, accompanied by a recommended action plan.
What if pausing non-brand search campaigns in the e-commerce example failed to yield a significant impact on performance? The data should unequivocally demonstrate this, and the advertiser should feel empowered to explore the underlying reasons. Perhaps the campaigns are targeting the wrong keyword themes, or maybe the advertiser is already saturating the market through other tactics like Performance Max, Meta, or AI-driven ad platforms, preventing non-brand search from breaking through the noise. Regardless of the cause, investigating the "why" opens the door for further testing and optimization.
Action plans are built upon insights derived from less-than-successful initiatives. This underscores the value of developing an ongoing testing framework rather than treating each test as an isolated, one-off event. This iterative process of testing, learning, and refining is fundamental to continuous improvement.
Incrementality Testing as a Sustained Practice
Incrementality testing is not a static exercise aimed at finding a singular, definitive answer from a single snapshot in time. Instead, it should evolve into an ongoing practice that consistently enhances the effectiveness of advertising accounts. In today’s complex omnichannel environment, no single system perfectly measures the effectiveness of every channel. Nevertheless, a robust incrementality testing framework cultivates confidence, ensures strategies remain fresh and relevant, and, most importantly, directly links paid search efforts to overarching business objectives. It can rightfully serve as the cornerstone of an effective account management framework. Businesses do not engage paid search managers or agencies solely for improved click-through rates; they seek a partnership that will demonstrably benefit their business. Incrementality testing emerges as the crucial mechanism that validates this fundamental assumption, transforming theoretical potential into measurable impact.
Lina Irawan
