The Measurement Gap: iROAS, Geo Holdouts, and Progressive Truth

[00:00:00] Luke: So the problem that exists as it relates to marketing measurement is that there's a measurement gap. Every brand really wants to understand the causal relationship between the advertising dollars they spend and the revenue they realize as a result. This is the central question of modern marketing.

[00:00:16] Does the spend actually work?

[00:00:20] Richard Gaffin: This episode of the E-Commerce Playbook is brought to you by Omnis Send. If you're serious about growing your brand through email and SMS marketing, Omnis Send has your back. It combines powerful automation, smart segmentation, and realtime performance insights, all in a platform that's intuitive and built specifically for e-commerce With prebuilt workflows, dynamic signup forms, and full multichannel capabilities, Omnis Send helps brands scale without unnecessary complexity.

[00:00:48] Join over 125,000 eCommerce marketers who rely on Omnis. Send every day. Head to omnis send.com/ctc and start on the free plan today. That's omni send.com/ctc.

[00:01:02] Luke: Hey, everyone. Welcome to CTC's Canon series, where we are walking through our methodology across the core disciplines, the CTC methodology canon, how we approach forecasting, modeling, measurement, meta ads, Google Ads, email strategy, creative strategy and the data that has informed the methodology across these disciplines.

[00:01:26] We're deep diving into each one. This is part three on measurement. We've walked through forecasting. We've walked through modeling. We are now on measurement, and then we will be sequencing into the other disciplines of creative strategy, meta, Google, email and on down the line from there. So Measurement is today's topic, and this may be one of the topics that gets the most heated debate outside of meta ads, and then creative strategy will be up there as well.

[00:01:56] So we have some hot topic conversations coming up in these in, in these next three conversations that we'll have as we walk through our series. But marketing measurement, this is how CTC closes the gap between what your data says and what's actually happening through incrementality testing, geo holdout studies, and commitment to moving closer to reality over time.

[00:02:16] We could spend the next two hours just walking through all the different frameworks and approaches to measurement that exist in the direct-to-consumer e-commerce space and the brands that we work with. Across our data set of hundreds of brands, we are able to see and get insight into the, into the conversations, the organizational structures, the platforms, the frameworks, the spreadsheets, the tools that are used are-- to surround this conversation of marketing measurement.

[00:02:40] From creating a weighted triangulation of marketing measurement based off of J-last click, an Adobe signal and Northbeam to using Haus or Measure as a source of truth, and then translating those to the platform ROAS targets to we're gonna use platform attribution as the key source of truth.

[00:02:57] On down the line, there's, there are an infinite n- combination of how brands are approaching marketing measurement. And our role and our responsibility is to identify what we see as being the most helpful framework for approaching this conversation that leads to the best decision-making in pursuit of the business hitting their business objective which is rooted in the first ser- part of the series around forecasting, and us aligning on what that core objective is.

[00:03:22] So the problem that exists as it relates to marketing measurement is that there's a measurement gap. Every brand really wants to understand the causal relationship between the advertising dollars they spend and the revenue they realize as a result. This is the central question of modern marketing.

[00:03:38] Does the spend actually work? A great measurement system, we believe, closes the gap between reality and f- and fiction when it comes to interpreting that effect. The wider the gap, the worse your capital allocation decisions, the wider the error bars are. The narrower the gap, the more confidently you can invest towards growth.

[00:03:58] So on-- If we picture two ends of a line, we have reality, the true incremental impact of your spend. We have fiction, what your platform dashboards report or whatever the tools are, and there's the gap in between those things. The measurement ex- system exists in that gap. The goal is not perfection. The goal is to mor- to move fiction closer to reality over time with increasing confidence.

[00:04:22] So some first principles to ground us in this conversation. Before building any measurement system, there are foundational truths that must be accepted. There aren't opinions, there aren't there con-- these are constraints that govern any honest approach to marketing measurement. So the first of these three principles is that media efficacy is in constant flux.

[00:04:43] The relationship between your ad spend and its revenue impact is not a constant. It changes day to day, week to week, driven by force- forces both within and outside your control. Any system that treats this relationship as fixed is lying to you. The second principle: You are always building an approximation.

[00:04:59] At all times, your measurement system is an attempt to build the closest approximation to reality that you can. There's no perfect measurement. There is only less wrong measurement. And the third principle: Progressive truth is the mechanism. Our approach is to build progressive truth. We wanna move closer and closer to reality over time.

[00:05:17] Every test, every data point, every experiment reduces the error rate of the system. Truth is not discovered in a single moment. It is accumulated over time and through reps and signal So there are many forces at play when it comes to thinking about the efficacy of the media spend which is really shaped by two categories of forces.

[00:05:37] And understanding the distinction between these two categories is critical to interpreting any measurement result. There are forces within your control, and there are forces outside of your control. So what are the forces within our control? The quality of your creative, campaign optimization, placement strategy, audience targeting, budget allocation, the landing page design and experience and we could go on and on.

[00:05:58] Forces outside of our control: general market demand, the macroeconomy, cost of ad inventory, platform algorithm changes, competitive intensity, seasonal demand shifts. So it's important for us to understand from a first principle standpoint that there are three core principles. Media efficacy is co- is in constant flu-flux, you're always building an approximation, and progressive truth is the mex- mechanism.

[00:06:22] And then to understand that within that dynamic, there are two categories of forces at play, those within your control and those outside of our control. This common understanding of the grounding of these principles then allows us to start to discuss the method by which we are going to build a r- the best approximation between these two points, given the principles we've, we've agreed upon and the forces that are at play.

[00:06:46] The m- the best mechanism for building progressive truth is through incrementality studies, specifically geo holdout tests. These experiment designs are the gold standard for isolating the causal impact of advertising spend on revenue. From a very high level geo holdout tests are where there are test regions in which the in which the marketing is being delivered and the impact is being measured.

[00:07:12] And then there are the control regions which are not receiving marketing for the channel being measured. And what we are measuring is the revenue dis- difference between the test and control regions relative to the synthe-synthetic control and the baseline that is established between those things.

[00:07:28] That helps us to understand what the true incremental lift of the measurement is. So geo holdout tests are identifying on a state, DMA GMA level, whatever the mechanism is to isolate specific geographies. Those are selected through a lot of smart data science to be able to understand the groups of those geographies that are going to be best representative and tied to the revenue impact for the business.

[00:07:52] But they're isolated, the impact of running marketing within a subset of regions versus with not versus not, then gives us a high confidence level causal relationship between the marketing spend that we are deploying on a specific channel or tactic and what happens when we remove that from the media mix.

[00:08:14] So, this measurement of the the, the media spend and the incrementality study geo holdout lifts need to be measured across every point of distribution as well. So the incremental lift should ideally be measured through not just your own dot-com because your advertising bleeds across channels. So you have your dot-com, you have Amazon, and then you have retail.com, and then in store and on down the line for additional distribution channels.

[00:08:39] So, an incrementality test gives you a snapshot of the causal effect of that revenue with some degree of confidence for that period of time. It's critical to recognize that this is a single data point that may or not-- may or may not be replicated again, right? Running a specific test at a specific point in time for a specific channel yields a specific result at a specific point in time.

[00:08:59] And when you're-- if you were to try to replicate that test, even within the same channel that you are isolating, similar budget levels, you are likely going to get a different result because that is at, is at a, it is at a different point in time relative to the prior test. So this is where the, the concept of pr-progressive truth becomes essential.

[00:09:18] A single test result is valuable but inefficient-- insufficient. The power comes from accumu-accumulation of these test results. So, let's deep dive into a geo lift study a bit more so that we're we share a common understanding of what that, what that looks like. So we design and deploy these tests through Statlas data science platform powered by the data science team, which provides end-to-end manage-end-to-end management of geo lift studies, from rec-recommendation to specific test outputs, validations, and results.

[00:09:51] What we look is the the power curve of these of these studies. We look at the results and we get them to a level to where the confidence level is at ninety percent confidence or better, so a p, a p-value of of p-- point one or less, so that we have a high confidence value against the revenue impact.

[00:10:08] And then again, we measure this across every point of distribution that we have data for, .com, Amazon, retail.com, as well as every c-core customer cohort. So new customer, new customers and new revenue, returning customers and returning revenue and on down the line so that we have a very clear understanding of the media's impact in each of the specific areas and each of our-- and the impact on the individual customer cohorts.

[00:10:33] So when we think about the framework, we have the-- We're grounded in the principles, we're grounded in the methodology, so incrementality, geo holdout tests. So now we need to build the framework for how we deploy this into an actual operational execution that leads to action that we can take based on the input.

[00:10:51] So we wanna build the largest database of test results, both in aggregate across all brands and for individual brands to con-continuously reduce the error rate of any presently applied measurement system. So stage one is that we start with an aggregate benchmark across all tests ever run and apply that against the brand's platform-reported revenue.

[00:11:13] So we have a database of hundreds of incrementality tests that we run across various platforms and tactics for a number of brands. And that gives us an ag-aggregate benchmark that gets us very close in many cases to what the, what the incrementality factor should be for for each of these platforms.

[00:11:30] And we start with that approximation as the best source of truth, right? Prior to us even running the incrementality geo holdout test, when we start working with a brand, we use the aggregate benchmark result from our database of incrementality tests for that specific channel and tactic to get us closer to what truth is in terms of that, that channel's impact.

[00:11:49] Richard Gaffin: Still juggling multiple tools for email SMS and Push Marketing. Omnis Send brings them all together in one powerful platform built for e-commerce, you can launch prebuilt automation workflows, built audience segments based on shopping behavior, and even gamify your signup forms without spending weeks learning the tool.

[00:12:08] It just works and it's built. To scale with you. Every account starts on a free plan, and unlike other platforms, Omnis Send gives you 24 7 human support no matter your tier. Check it out@omnissend.com slash ctc or click the link in the description.

[00:12:26] Luke 2: So that's stage one, is using the aggregate benchmark to inform what the incrementality factor should be for that specific channel and tactic. Stage two is that we are going to get the first test result back from an incrementality and geo holdout test specific to that brand, so-- and that channel and tactic.

[00:12:44] So we run a geo holdout test specific to that business, get the factor, and then what we are going to do is that we are going to weight that individual result relative to its confidence level and relative to a relationship to the aggregate benchmark that we have. We're not gonna move towards it all the way but we are going to get closer to that approximation of what the impact is.

[00:13:05] 'Cause again, that test result was at a specific point in time in a specific structure, and so there should be some impact to the aggregate benchmark as it results to weighting that. And then stage three is that as more test results come in over time, the median that re- that's of the set represents the measurement with the lowest error, right?

[00:13:22] So we, we continue to get more data signals and move towards to this progressive truth. So if you were to visualize what this looks like in practice, imagine a scatter plot of incrementality test results over time. So each dot on a scatter pro- plot represents a single test. Early on, the dots are sparse and widely scattered, right?

[00:13:41] So the confidence in any, any single test result is low. But as tests accumulate over over a longer period of time, patterns emerge and the median conversion converges. Error shrinks against that, and we can get higher and higher confidence in terms of that channel and tactic, tactic's true impact. So the starting point for any brand should be the aggregate benchmark.

[00:14:01] The second an individual test result arrives, we weight it relative to its confidence and shift towards it. With each subsequent test, the system gets less wrong The progressive truth in practice, CTC maintains one of the largest proprietary database of incrementality test results. It's not theoretical.

[00:14:17] It's a real geo holdout test run across real brands with real dollars. We have hundreds of tests across a large number of channels as well and various levels of confidence in those results as well. And what that leaves us with is channel-level benchmarks based on this aggregate data set that we can use as a starting point to move a brand immediately towards higher confidence in terms of their media's impact on that specific channel or tactic.

[00:14:41] There are factors that we use that represent the relationship between the platform-related reported revenue and the and the actual incremental revenue from each of these channels. And these vary. To name a few, Facebook acquisition campaigns the median iROAS is a one point one four across our data set.

[00:15:01] YouTube, it's a one point one. Google Ads non-brand, point six seven. Facebook non-acquisition, point six oh, and on down the line. But we have the database across hundreds of stores, test results that where we're able to understand the iROAS, the incrementality factor and then understand the distribution in, in each of these results as well related to this channel.

[00:15:22] The, the data based on our aggregate set of benchmarks, though, tells a really clear story, which is that Facebook acquisition, a median iROAS of one point one four, is the most reliably incremental channel. A second trend is that Google-branded search, median iROAS of point two seven, confirms what the theory predicts.

[00:15:40] Platforms dramatically over-report on last-click channels or channels that are that are very close to the bottom of the funnel in terms of capturing demand. And the ranges show why single test results are insufficient. Facebook acquisition ranges from a point five x to a two point four x and on down the line.

[00:15:56] So these factors are not permanent truths. They are the current best approximation based on the aggregate of all available at evidence. As new test results come in, every number in this table will be refined. That is the system working as designed. As we've mentioned, a single test result is a snapshot, valuable but incomplete.

[00:16:17] And so the principle of always-on testing is important for brands to adopt in terms of thinking about measurement, that this is not a one and done, but that we are going to transform that snapshot into a distribution of potential incremental outcomes for each individual business. So as each indiv- individual test result accumulates for a brand, the bounds of that distri- distribution represent the total possible error in either direction.

[00:16:41] The median represents the point at which we are at any given time most likely to predit- predict the future outcome. So always-on testing builds a distribution of outcomes. The median converges towards predictive accuracy. This is a practice of measurement that once aligned on the principles and the approach is going to be a continued practice and getting more and more data points to get higher and higher confidence in the, the true impact of our media media measurement.

[00:17:10] As the database of test results grows, the system becomes increasingly sophisticated. Seasonal effects, sale moments, and other variables can be incorporated. Rather than applying a simple median across all results, the factor can be adjusted based on the conditions that most likely match the present moment.

[00:17:26] What this enables operationally is apples to apples comparison across all channels. So this is one of the most powerful benefits of this measurement systems is that it enables true like for like comparison between channels and ad products. So without normalization, comparing meta acquisition spend to Google branded search is meaningless.

[00:17:43] The platforms report in different attribution windows with fundamentally different relationships to incrementality. Google brand search is a well-understood example of this. Google Ads by default reports on a on a 31:1 or 37:1 attribution setting. Branded search will dramatically over-report its return on ad spend.

[00:18:00] Branded search is a final step action for many shoppers on their path to purchase. The customer's already going to buy. They typed your brand name into Google, clicked your ad, and completed the purchase. The ad gets credit, but the purchas- purchase was still likely to have occurred had the ad not a- appeared.

[00:18:15] So the factors of channels living at different points in the customer purchase journey and different platforms using different attribution settings makes it very challenging to compare performance like for like and make clear investment decisions. Once we adopt an I- an incrementality and iROAS practice, we're able to normalize across all channels so that the platform ROAS on Meta may read a 3.2X, but based on the incrementality factor, the true iROAS is a 3.7X.

[00:18:43] Then on Google, our platform ROAS is a 12.5X, but based on the incrementality factors and benchmarks the iROAS is a 3.1X. So now we can look at what was previously a 3.2 on Meta and a 12.5 on Google branded. We can look at as a 3.7 and 3.1 and look at those numbers like for like and make investment decisions accordingly.

[00:19:03] So by running incrementality studies on every channel and computing the incrementality factor, we create a normalized iROAS that allows us and the profit engineer or the brand to compare the performance of their media channels on a like by like basis, and this is what enables real capital allocation decisions.

[00:19:21] These incrementality factors are not theoretical. We operationalize them into CTC Statlas platform through the MMM roadmap, which uses tester-derived iROAS to prioritize channel allocation for each brand. So each of the most up-to-date test result factors live behind Statlas. Every ad channel, campaign and tactic are applied against their respective factors so that everywhere that we are looking at data, from the daily forecasting and the dashboard to the tracker tabs for each of the media channels, everywhere we're looking, we're looking at the normalized incrementality adjusted revenue and ROAS for each of these channels that allows us to be able to make the most impactful decisions.

[00:20:08] To wrap this up, iROAS is subordinate to reality. Even a well-calibrated incremental return on ad spend should always be subordinate to the realities of revenue and contribution margin at any given time. We should always assume some amount of error in the system. So when iROAS and actual business outcome are incongruent, meaning one is moving up, your iROAS looks really good, while the other contribution margin is flat or moving in the opposite direction, that is the signal to examine the underlying measurement system and recalibrate.

[00:20:37] Contribution margin and the business outcome lives at the top of the assessment pyramid. iROAS is, is subordinate to that reality. If iROAS is improving, revenue grow- is growing, and contribution margin is expanding, that's a signal that there's congruency. If any of those signals have adverse relationships to one another, then there's incongruency as it re- as it relates to that impact as well.

[00:21:05] One of uni- one of the unique things that CT- CTC brings to the table is the ability to design, deploy, and report on geo holdout tests as part of our core service offering. But the real dif- differentiator goes further than this. We have the obligation to operationalize those results in your ad account and to bring those effects to life in our decision-making and reporting.

[00:21:23] So rather than being a standalone measurement platform or tool that where we design and run tests, deliver a report r- with results, maybe suggest actions, and then stop there and hand off to you or your other partners or your team, what we do as part of the profit engine and the profit engineer sitting inside of that is we design and run the tests, we deliver the results with interpretation, but then we operationalize those in the ad account.

[00:21:46] We calibrate the cost controls, the bid targets, we report on the realized business outcome, and we're accountable to the business objective at the end of the day, and the suggestions and test results that we're making need to impact those in a way that leads to the business outcome that we are after.

[00:22:05] So in conclusion, our measurement philosophy involves building an ongoing roadmap toward progressively better truth, one that allows to get us closer and closer to the approximation of reality and the causal effects of our media at any given point in time. The best measurement methodology will include a constant pursuit of new data points, a period application that builds, not undermines, subordination to business reality, end-to-end accountability, dynamic benchmarks from aggregate intelligence, and honest treatment of uncertainty.

[00:22:35] The measurement system is only as good as its ability to lead to a better allocation of cra- capital across the available media channels. That is the standard. That is what we measure ourselves against, not the elegance of a model, but the quality of the decisions that it produces. This was CTC Canon series on measurement.

[00:22:53] Looking forward to seeing you in our subsequent parts of going through the CTC methodology canon