The Modern Media Playbook

[00:00:00] Taylor Holiday: . Alright, welcome back to the another episode of the E-Commerce Playbook podcast. Today we are sort of showing our homework. We are taking you inside the development of a system that we have needed to get clear on for ourselves because the world has changed so much over the last few years as it relates to how you think about and manage a media buying system on the behalf of.

[00:00:23] Taylor Holiday: D two C brands in pursuit of profitability. There have been the introduction of a immense focus on, incrementality and testing and experimentation at the center of the workflow. And it, I have found that it is actually more challenging than people realize to be able to turn experimentation, a budgeting and forecasting process, media man or media mix modeling and trying to decide about channel allocation into one consistent workflow.

[00:00:49] Taylor Holiday: And so Luke, who is, you guys know him very well is here to talk about what he has been working on, which is. The defining playbook for how CTC builds a [00:01:00] modern media management system that covers all of these elements to build a day-to-day way to actually execute against the ideology that's espoused all over the place.

[00:01:09] Taylor Holiday: So if you have been wondering about how can I turn the idea of incrementality or MMM or fp and a. Into a way to actually day to day execute your media playbook. This is the podcast for you, and there's an awesome blog that goes with this that we have now published on our website that will be linked in the show notes that you can go and download and absorb for yourself.

[00:01:30] Taylor Holiday: We're gonna turn this into a sort of PDF style deck as well that we're gonna be using to train our internal team as well as update our customers on. But Luke, welcome back to the show and why don't you give us a little bit about. How you approached or what you've been seeing that led to the need for us to create this documentation?

[00:01:47] Luke Austin: Yeah, so I think the point of view we have is we, we see hundreds and hundreds of direct to consumer e-commerce brands in a pretty wide revenue range on a daily basis, and we have these [00:02:00] conversations consistently, and so we're able to see. A lot of different ways of being for direct to consumer e-commerce orgs and how they approach organizational structure, media management, the measurement systems they have in place, what tools and systems they have built out for forecasting, fp a like you mentioned.

[00:02:15] Luke Austin: Right? And so within that context, like we're able to have the insight and we should have the leverage of being able to. See what's working well in that ecosystem and what, what's not working well and sort of assemble what we believe to be the best framework based on hundreds of direct consumer eCommerce brands.

[00:02:32] Luke Austin: I think if we're not, if we're not doing that effectively, we're, we're, we're not playing our role well as, as a partner in, in what we're in, what we're thriving. So what a big piece of that relates to. The conversations, the hours we spend talking about media measurement and budget allocation and then day-to-day optimization against that.

[00:02:51] Luke Austin: And so we'll kind of get through the, the sort of the core questions that live, live under this. But over, over the years, we have had, this is constituted a meaningful portion of our time as [00:03:00] it has likely for anyone listening. It accounts for a substantial portion of your time on a daily, weekly basis is conversations around this topic. And so in an effort to help. Bring clarity to what we see working well and what we've seen not work well. We've worked hard to sort of clarify the framework that what, what are the key pieces of a framework around measurement and a system for operationalizing against measurement and, and optimization that we can employ here at TTC on behalf of our customers, and then share with, share with you all as well.

[00:03:31] Taylor Holiday: Yeah, so I, I really think we've talked a lot about this, the idea that this is what an agency's value is, as much like the reason you go to a manufacturer to make your product is because they have spent years developing. The tooling and systems to be able to produce that as efficiently as possible. The reality is that media management or growth marketing is the same thing.

[00:03:47] Taylor Holiday: It's a system of effort that we have spent many years working through reps to get right. And so what we're gonna share today is a little bit about the core elements of our system and how they get deployed on behalf of our customers. [00:04:00] And this, it's something that you could take and think about how you could apply some of the principles yourselves, or can be looked at as an evaluation of our system and determination if, if it would be useful to you.

[00:04:10] Taylor Holiday: So let's start with the first point. 'cause I actually think this is an area that might be a little boring, but it is unbelievable to me still how poor. The underlying quality of the information is that most people are making decisions with. So let's start with the foundational premise of our system, which is data integrity and data integration across multiple silos of information.

[00:04:29] Taylor Holiday: So you call it the three necessary data dimensions for any underpinning of a system that you're trying to develop. What are they and why are they so critical to the start of this process?

[00:04:39] Luke Austin: Yeah, so everything that we'll discuss here in in each of these buckets, ties, ties to the pillars of. Budget allocation and measurement and, and sort of the three, just to take one step back to you, like the three main questions that we all have to answer and, and and have conversations around on a consistent basis is what should the total budget allocation be?

[00:04:57] Luke Austin: How should I allocate that budget across channels? And then [00:05:00] what, what system do I have for operationalizing against against that budget allocation, optimizing against it on a day-to-day basis? How do I actually go and execute execute on those, on those inputs? So in service of the, that, that bucket of. This, this first section around data integration is, is the first step to be able to get there. And, and I think what we've seen is that, in this process, we have seen all the tools, all the processes, internal tools built out that have triangulation from seven different inputs, from seven different tools, all the way down to we just use, you know, Adobe, click Ross and everywhere in between and. The most important thing in related to data integration and what data you have available, is that any platform that you are using or system that you're using to answer these question on budget allocation measurement. have data integrated across these three dimensions. One is revenue. So this is from, you know, your Shopify online store [00:06:00] transactions.

[00:06:00] Luke Austin: So it's gonna be your revenue and orders from your online store. Two, marketing investment meta, Google affiliate directly from those platforms. And then three costs, your product costs, your delivery costs, fulfillment, et cetera. Payment processor fees, all the costs, the variable costs that go into making.

[00:06:17] Luke Austin: That revenue possible and making those orders happen. So revenue, market, investment and costs those are the three necessary data dimensions and it's worth noting that the system doesn't improve if you add in more dimensions, right? If you have these three dimensions plus six other dimensions, it doesn't improve the output.

[00:06:36] Luke Austin: It actually con convolutes it, I think. It could be something we can unpack further, but these are the three necessary dimensions and the three main dimensions that you should be looking at to help serve these questions. Revenue, marketing, investment, and cost. And I think. one example to frame up why this is so important is in, in the paper we look at is Google's budget and bid simulator.

[00:06:56] Luke Austin: And, and meta has a similar thing for certain campaign types, right? Where you can go in and you [00:07:00] can look at, at different levels of investment. If I increase my budget budget, what is the anticipated increase in conversion value from Google conversions? And sort of look at that, look at that curve, right? And understanding that there is a bias, of course, that exists for Google and meta in recommending the additional budget allocation against these things. think we, our perspective would be they have the best insight into what the diminishing curve is related to the increased spin level and the conversion value.

[00:07:25] Luke Austin: Right? 'cause they, although there is bias on them to increase ad investment, if they make a suggestion for increase your budget 500 bucks and the conversion value is wildly off, then they're. Tool prediction that erodes trust in the tool, which is gonna have long, long-term impact. So I think they have helpful, really helpful insight into these things.

[00:07:41] Luke Austin: But they don't have the third input. They have revenue and they have marketing investment. They don't have the cost input into, into that matrix. And so what you'll do is if you, you can go look at this from one of your campaigns. Look at one specific campaign, increase the budget $3,000, $5,000, look at the increase in conversion value in [00:08:00] orders, and you're gonna see some increase against those dimensions. But then take that data, pull it into a spreadsheet, overlay your cost of goods and your and your delivery costs to get into contribution margin at those different levels of spend. And you'll see a totally different optimal point of increased budget recommendation versus what Google's recommending. And it's because they don't have the cost level data that's so necessary.

[00:08:21] Luke Austin: So revenue, marketing, investment, and costs are the three necessary dimensions for data integration for any tool to even be helpful in starting to answer these questions around budget allocation. Medium measurement and optimization.

[00:08:34] Taylor Holiday: Yeah. And, and this, this, I want to just highlight that this is probably sounds obvious, but. I'm just gonna tell you, we work with hundreds of businesses and the percentage of brands that come to us with clarity across their data in these three dimensions is less than 5%. Like it is it, it is unbelievable to me how fragmented and poor the structure [00:09:00] of this data is inside of businesses.

[00:09:02] Taylor Holiday: And if we just go through all three of them real quick to illustrate what I mean, so revenue, this sounds easy, but organizations don't universally know what their revenue definition is. So what do I mean by revenue definition? The distinction between gross revenue, net revenue, order revenue, total sales, and the presentation of different BI tools gives you different views that fundamentally alter the way you think about.

[00:09:21] Taylor Holiday: Your revenue generation in particular, the biggest challenges around the idea of net revenue and the inclusion of returns or not, and whether those returns are accrued or in real time for your business will fundamentally alter your view of revenue definitions in different periods of time, and significantly alter then the relationship to your media performance over time.

[00:09:41] Taylor Holiday: And so there's a lot of work to be done. Al also, like in Shopify, some people include. Third party revenue from a wholesale channel or fair, or they, maybe they have a point of sale device from their one store and they, they will or won't include that revenue. So the source is Amazon included? Is it not getting clarity [00:10:00] of how much money did we make and what is the definition of that?

[00:10:02] Taylor Holiday: Revenue is actually a lot of work and to create consensus across an organization, a lot of work. Number two is marketing investments. And this is one where. This gets challenging because there are the obvious channels that have direct API connections where you're getting a real time view of costs. The metas, the Googles, the Snapchats, the tiktoks, the Pinterest.

[00:10:19] Taylor Holiday: But then there's a whole slew of things, let's say podcasts or affiliate or some people wanna include things like influencer payment. And so there's this question of, again, cash versus accrual idea of, okay, so you paid an influencer a thousand dollars on a Tuesday and they post it on a Wednesday. When does that cost show up on the daily realization of revenue?

[00:10:39] Taylor Holiday: Is it amortized across a period of time? Is it shown up in a single moment? All of these questions are unanswered and ambiguous and different within every organization, and that's before we even get to the most complicated one, which is the cost. The reality is that COGS is an idea, it's not an objective reality.

[00:10:56] Taylor Holiday: And in many organizations will use FIFO accounting [00:11:00] first in, first out. For what the cost of goods is of any unit sold, because the reality is the unit costs change over time. Tariffs are an illustration of this in the most obvious fashion, but there's also things related to shipping or shrinkage or all other elements that affect brands, underlying unit cogs that change and aren't reconciled regularly.

[00:11:17] Taylor Holiday: Then there's fulfillment expenses and delivery fees that, again, a lot of people have a three PL partner where that's not, they don't have a real time view into order costs, and the reality is, here's another funny fact is that your actual shipping costs. Don't get realized in real times. There's adjustments to those costs all the time Act based on the actual delivery fulfillment expense.

[00:11:37] Taylor Holiday: So it is actually very complicated to get to root truth here or to get to something that we believe is accurate enough to build a foundation for good decision making. And I really believe that one of the big future things that's gonna happen in our industry is that there's gonna be a lot of investment into data warehousing and data organization because it is unbelievable to me how bad the information is that most people are making decisions on.

[00:11:57] Taylor Holiday: And this is before we even talk about building a system. [00:12:00] The question is just what is the information that we're looking at? And it is still very, very messy.

[00:12:05] Luke Austin: Yeah. And, and for the marketers listening, this, this may seem like three levels too deep in the conversation where we're getting by just fine looking at spend and revenue. MER and a MER, right? Like the, the, the four, the four core metrics. But what, how this actually plays out is any one of those examples.

[00:12:23] Luke Austin: For example, if returns are processed every Tuesday and Thursday in big batches and you're expecting, you know, 13,000 in returns in each of those days and you have zero the rest of the days, how are you're accruing and accounting for those as you get to the end of the month? you have a Tuesday and a Thursday and the Thursday's the 31st of October, right?

[00:12:41] Luke Austin: And or if Wednesday's the 31st of October, that's $13,000 difference in contribution margin, directly unrelated to the marketing investment that ultimately the, the business outcome, the enterprise value is going to be based on. And the finance department and the board, how the targets, that's what they're looking at, right?

[00:12:57] Luke Austin: That's what they're interested, not what the MER lands out for the [00:13:00] month. That is not how the business is going to be assessed. And so the opportunity you left on the table by not pushing additional dollars because the month landed on a Wednesday instead of Thursday, so you could be a little more aggressive or inversely, you didn't account for the additional returns that were gonna be processed on that final day of the month.

[00:13:17] Luke Austin: And so you overspent and the month landed below the EBITDA expectation. Is go leads to the kind of conversations that we all are engaged in related to how we land against the, the, the business expectation, which is based on what the EBITDA outcome is, is against the defined time period. And so each of these things are critical to understanding how much you can push and pull the business in, in different timeframes and make sure there's nothing left on the table.

[00:13:43] Taylor Holiday: And, and this is what great marketers actually decide to take responsibility for, is they understand that business is a, is a, it's a 12 it's a series of 12 periods over a year that are all their own individual game that you're playing. And you have to win all of them. Or you have to win as many of them as possible.

[00:13:57] Taylor Holiday: And so really the more you understand all the [00:14:00] ways in which the game gets reconciled in which the scoreboard actually is set, the more impactful you become as an individual leader and the more authority you can take over the outcome versus being subject to like, I didn't know that that was gonna happen to returns.

[00:14:12] Taylor Holiday: Or we thought X, Y, and Z. Like that is one of the big distinctions of where we really. I, I press our people all the time 'cause they wanna fall victim to this too. But the more responsibility and authority you claim over the knowledge and the outcome, the more impactful you can be within the organization at the highest levels.

[00:14:30] Taylor Holiday: And so I think that that is really the key is that to understand at the end of the day. A CEO who has to go report to the board doesn't get to use the excuse of, I didn't know or I was unaware. And so if you want to impact that person and somebody be somebody that they can trust, you have to take accountability towards all that information.

[00:14:45] Taylor Holiday: It means digging deeply into the roots of why everything the way is the way it is. And we discover things all the time. So. I'm gonna get off the soapbox now 'cause we gotta move on. But the point is, is that the underlying data integrity is the foundation of this system and you're probably not [00:15:00] spending enough time on it.

[00:15:01] Taylor Holiday: But, okay, so we have, let's assume now that we've solved for that, we've got visibility into data integrity, Luke, and now if I think about a media plan, the first thing we have to decide is how much money do I get to spent? What's my budget? And this is a process that I think, again, how people are setting media budgets.

[00:15:17] Taylor Holiday: Some are trend forecasts, some are just estimations. But we have spent a lot of time in trying to think about a specific mechanism for setting a media budget in each month relative to the financial goal that serves as the foundation for the dollar amount that we're using to spend. So talk a little bit about our budgeting process and how we go about setting that number.

[00:15:38] Luke Austin: Yeah, so. The core point around budgeting, determining your total budget allocation in any given monthly or yearly timeframe, is that every business has an efficiency degradation curve that's unique to them specifically, and there's a variation in your efficiency degradation curve at different times in the year.

[00:15:56] Luke Austin: So that that is something that we have seen [00:16:00] across hundreds of brands, that that dynamic exists for every. Business. And it is at the basis of that understanding that we need to have a system that is able to account for that change in seasonality, the change around holidays and marketing moments, and then the historical degradation of efficiencies specific to your brand and what we've, what we've seen most consistently in terms of budget allocation. Is in the better case scenarios, there is a, a, a budget allocation is sort of like a variable against a fixed input related to MER or ACOs, right? And so it allows flexing up and down against a blended marketing metric like add to sales ACOs or, or MER, many other cases, there's a fixed budget allocation that's predetermined, maybe a quarter in advance or maybe even at the beginning of the year.

[00:16:48] Luke Austin: That's the budget for the month that gets set. That's sort of the least helpful in, in this process, at least with a a flexible budget allocation. It gets me a cost. You're sort of taking advantage of those. those things. But [00:17:00] the every month has a different degradation efficiency curve. And so aligning against the same metric, A, we need a 20% ACOs or a five MER, that's our goal.

[00:17:09] Luke Austin: Using that as the thing that we're holding in any given time period doesn't account for that. You could be pushing more in certain time periods. You should be holding back more in the budget in other time periods. Doesn't account for variations in discounting and margin in different, in different time periods as well.

[00:17:24] Luke Austin: Right? We have a marketing calendar that's more aggressive discounting in November, and so your MER should be higher relative to other time periods. And so this is sort of the basis of what we've seen be helpful and unhelpful, and the reality of every business having an efficiency degradation curve unique to them, that changes throughout the course of the year.

[00:17:41] Luke Austin: And so. What we've done within the understanding of seeing this is built out what we call a spending power model. Many of you are probably familiar with it in the conversations over, over the recent months, but the spending power model is an ensemble model. It creates, it's built on [00:18:00] using 30 different time series models that look at things like seasonality, historical degradation of efficiency. Holidays and marketing moment impact for the brand. Historically, we'll even look at categorical and competitive search trends. So we, we overlay 30 different time series models to be able to back into the best expectation for the current state of the business and, and understand what that efficiency degradation curve looks like on a monthly period, and then pick the optimal point against different optimization selections we have for those time periods.

[00:18:29] Luke Austin: And, and this is the, the most important thing, which is. Tying it to the first pillar that we discussed, which is is a specific business optimization. There's a specific goal for the business at any given point in time that relates to a financial outcome that isn't MER or ACOs, MER and ACOs are never the goal for a business at any given point in time. They're typically used as a constraint set by. Finance minded folks to help constrain marketing against an outcome that we [00:19:00] will believe will get us to the business objective over that time period. What we're much more interested in is building a system that allows us to select the business optimization that we want for that period of time, and then allow that to dictate the budget based on, again, the specific business and the specific time of year, which suspending power allows us to do and, and allows us to look at different business optimizations. And then be able to allocate the correct budget for that time of year based on the business, the actual business objective, rather than a proxy metric that we're using to hold against the budget allocation.

[00:19:28] Taylor Holiday: So one, one of the things I wanna keep doing here is I want to say that we are. Doing two things at the same time that I recognize that you as a listener might struggle with. One is we're saying that each of these elements is incredibly complex. Okay? So we're saying data integrity, incredibly complex, setting the budget incredibly complex.

[00:19:45] Taylor Holiday: But I also want you to understand that that's part of why, the responsibility of us to turn that into a very simple tool that allows our team to go in and interact with. The curve that exists, and you'll see this image here in the blog article, Corey, we can sort of link to it as we're [00:20:00] showing. We've talked about our spending power models That allows a growth strategist to be equipped to go in and quickly understand the distinctions in the differences and make a quick decision is a very powerful tool that enables efficiency.

[00:20:12] Taylor Holiday: And efficacy of delivery of the system. So why is this so hard for brands to set budgets Well, because it is very complicated if you haven't spent many years and many reps understanding all of the inputs that affect your potential spend. We talked about seasonality, we talked about the marketing calendar, we talked about Google trends, we talked about all these things that allows us to build a version of what is most likely to occur to help you understand the choices of your future.

[00:20:34] Taylor Holiday: One of the most common questions we get asked by a brand are like, when we talk about media budget, it's like, well, what are my choices? If I spend more and make less, what does that look like? If I spend less and make more, what does that look like? Well, this is how we help them understand the options that are available to them Now.

[00:20:49] Taylor Holiday: Every model still requires that we go out and execute against the premise, but the starting point and the likelihood of accuracy matters here. And so this is an important building block to what is ultimately the day-to-day [00:21:00] execution. So I get it. It sounds complex and I'm not gonna minimize it. It is hard to understand all the things that affect your potential media budget, but we've built ways to make this simple, and that is where.

[00:21:11] Taylor Holiday: Just like your manufacturer, building tooling that allows you to move quickly to do something complicated, like make a widget with lots of small parts. The same thing is true here. The idea of setting media budget is complicated, but we've built tooling that allows us to move fast in the execution of it.

[00:21:27] Taylor Holiday: Alright, Luke. Another complicated thing. Okay, so you have a number, but now I have to decide how much do I spend on meta? How much do I spend on Google? It's Snapchat, Pinterest, TikTok podcast. Allocate. How do I actually decide the allocation of that media across channels now that I have a budget?

[00:21:43] Luke Austin: Yeah. So one thing I'll say to this, to this point, 'cause it's one of the, it's sort of the next piece of the system is. we're, this is our system defined at this point, at this moment in time. Based on the tooling that we have built and what we're seeing across, across our dataset of [00:22:00] brands. We are continuing to evolve all of our processes and methodologies based on the best available truth at that point in time and what we're seeing working.

[00:22:08] Luke Austin: So we're working on some things right now related to this this conversation. A budget allocation. Related to MMM integration and how that helps us to level it up, which would be cool. 'cause we, we've seen, we've seen ways that we can improve this further partnered with yeah, we, it won't go too far there that that'll be in, in the coming weeks here. But right now as it relates to budget allocation. Our perspective on budget allocation is that the budget should follow the efficiency expectation for that specific channel and not the other way around, which is an important sort of starting point foundational principle, which is we need to set what the business expectation is for that period in time, and then what the return is expected of that channel.

[00:22:46] Luke Austin: And then we need to go and maximize the opportunity against that specific channel, against that outcome. 'cause what we're not looking to do is say, here's your optimal budget allocation based on your historical. Performance, and you can spend $300,000 this month and that's it, [00:23:00] right? Like that's, that's the best that you can expect. And here's your budget, meta budget allocation. Don't spend a dollar over 180,000 this month. What we're, what we're looking to do is set what the efficiency expectation is that ladders up to the business objective, and then maximize the volume againsts. That's where that, so that we're not leaving any of the opportunity on the table.

[00:23:16] Luke Austin: And so our principle around budget allocation starts with that understanding, which is. What we do, we integrate the data that looks at revenue marketing costs. Then we set the budget allocation based on the business objective, maximizing contribution margin that month, maximizing first time customer revenue at breakeven contribution, and then that sets what the expectation is for the business in terms of dollar end to dollar out, return on invested capital for that time period. That is then how we are backing into what the efficiency expectation is for each of the channels with incrementality as the source of truth for that connection between, between those, those two things. So there, there's a law that we could discuss under the, under [00:24:00] the umbrella of incrementality, but the summary of our perspective here would be

[00:24:05] Taylor Holiday: This.

[00:24:06] Luke Austin: We've seen there's consistent consensus that geo holdout testing through experimentation is the gold standard of, of measurement and has been at some time. There's not, there's not much argument related to that point, and I think most brands say, okay, I'm bought in. I'm interested. What's been The challenge we've seen is two things.

[00:24:25] Luke Austin: One, tools not being available at a level that's cost effective for brands of for most of the brands of certain size, to be able to sort of, start working with incre fatality testing tools that you hold out and just being cost prohibitive in that way. And then the second challenge being that operationalizing the outcome of these tests is really challenging as well, because what you get through an MTA tool like a rocker box or North Beam, is you get a certain MTA model. And then it just factors out and you can see each of your channels in real time against that, and you can sort of optimize against them in that way. And it makes it really simple to operationalize it. [00:25:00] So it sort of suffices for that second challenge. But what an MTA tool doesn't do is suffice for drawing a causal connection at a high level of confidence through experimentation. What the, a channel's actual contribution to the business is, right? It's fractional credit overlaid against each of the channels based on the historical in a historical model. It's not done through experimentation to lead us to. A stat sig confidence level outcome of the channel. So it doesn't suffice for, for challenge number one.

[00:25:26] Luke Austin: So that's really what we're going for is how do we how do we bring incrementality to the, the majority of brands so everyone can engage in it? And then how do we operationalize that in the way that makes it accessible? But geo hold app testing. It needs to be the core of budget allocation as it helps to understand what a channel, what opportunity exists for a channel relative to its actual contribution to the business.

[00:25:53] Taylor Holiday: So one of the things, 'cause I think this is where there's sort of a chicken and egg game. [00:26:00] Between, do I have any tests? If no, what do I do pre-testing. So in other words, like, all this dependency that I've now produced a series of experiments that give me information versus what do I do prior to the existence of that information becomes sort of this tension for brands where they show up and they're like, okay, even if I agree that generally incrementality is the premise I don't have any test results and that's gonna take me months to produce a, a consistent set of information what do you do?

[00:26:31] Taylor Holiday: In light of that, and how does this process enable people to make the best decisions with the information that they currently have available, and how does it become progressively better over time?

[00:26:42] Luke Austin: Yeah, so the framework that we built to, to help solve for that is. Is two step one. We're using benchmarks based on our existing data set of tests. Again, if we have hundreds and hundreds of brands and test results for them. Those should be inputs that you're able to leverage for the, for the benefit of your business in some timeframe, [00:27:00] right?

[00:27:00] Luke Austin: So we use benchmarks in the short term. And then the second piece of it is that we help to prioritize your testing roadmap based on business impact. So lemme tease out both of those, both of those things a little bit more incrementality, benchmark. So you've never done GL holdout testing before. You have six different channels and tactics that you need to test between meta acquisition.

[00:27:18] Luke Austin: Meta retention, Google brand, Google non-brand, and then let's call it app loving and TikTok, right? So you have six different channels and tactics that you need to get measurement reads on. And there's gonna be some time lag against getting all of this. You can't run all six tests simultaneously, likely, unless you're spending an incredible amount of money on each at the same time.

[00:27:35] Luke Austin: So what we use is our incrementality benchmarks based on our data set of tests. As the starting point for you to determine what you should expect that channel's true contribution to be. So I'll give an example, meta acquisition based on our data set of incrementality tests from our incrementality tool, as well as looking at test results across some of the other tools that exist measured.

[00:27:57] Luke Austin: How's the other folks doing geo holdout testing. [00:28:00] We've aggregated these test results and we've seen that on average meta acquisition. The incrementality read against the seven day click reported meta acquisition on platform. 120% incremental. So if your meta on platform ROAS is reading 2.0 on a seven day click basis, the actual contribution to your business is likely closer to 2.4 120% of that of that 2.0 outcome. And we have that for each of, each of the other channels I, I described. And so what we can do is run with the incrementality benchmarks while we're doing the test to validate if the bi, if the incrementality factor for your brand is widely disparate from what the incrementality benchmark is for that time period.

[00:28:42] Luke Austin: So we're using, again, the best available source of truth at that time. And then we go into validating that result for your brand through in a priority roadmap that we built out. So we have, we have a tool in stats again, that we, what we do is we look at each one of your channels, we look at the amount [00:29:00] of spend, the amount of platform attributed revenue coming through each of those channels. And then we we apply our incrementality benchmark factors to the platform reported revenue from each of your ad channels to look at what is the best and worst case scenario, contribution margin from that channel on the, on the upper and the lower end. So we have a wide range, again, of incrementality tests all the way from. Meta acquisition is 250%. Incremental to meta acquisition is 50% incremental, right? And on average we're saying, okay, a hundred, 120% is a good starting point, but there is a wide range of outcomes. And brands in unique categories with unique, unique buying behaviors, they can, they can live at a different point in that spectrum.

[00:29:38] Luke Austin: And so what we look at is the worst case scenario being if your med acquisition came back at the low end of our incrementality test results, or if your med acquisition on the high end came at the, at the upper threshold of the results. What, what is the range of that risk that currently exists in your media mix relative to those things?

[00:29:56] Luke Austin: The amount of unaccounted for unmeasured. True revenue [00:30:00] impact from each of these channels. So we apply the upper and lower bound factors to each of your ad channels, and then we prioritize the testing roadmap accordingly. Right? If you're really interested in testing out YouTube demand gen, but if we're only gonna, if we're only spending $20,000 a month on YouTube, the amount of risk or opportunity that that represents to your business in the upper and lower bound is gonna be really minimal to compared to meta, which is spending $2 million a month, right? Or maybe it's Google non-brand, which has a, a more budget allocation and, and upper and lower and there.

[00:30:29] Luke Austin: So start with the incrementality benchmark factors. That's step one here, is that we are operating on best available source of truth. And then we go and validate each of those factors according to the opportunity or risk that each one of your channels represents to your business from a incrementality from an incrementality adjusted perspective.

[00:30:49] Luke Austin: And we go validate each one. And then when the test result comes back from meta acquisition, we update your meta acquisition

[00:30:55] Taylor Holiday: Okay. Okay, cool.

[00:30:56] Luke Austin: validated factor, and we

[00:30:57] Taylor Holiday: So I'm clear on that.

[00:30:58] Luke Austin: from there.

[00:30:59] Taylor Holiday: [00:31:00] Yeah. Th and this is where I think one of the deep illusions that gets perpetuated by people with an incentive to do so is a couple of different things. One is this idea that like everyone is unique and everyone must test individually to get to any useful piece of knowledge that can be acted upon.

[00:31:14] Taylor Holiday: And what I'll tell you is like that, that's just not true. Like that there is a level of precision. That can be improved upon over time, but the, the amount of increased precision potentially isn't that valuable. So in other words when we look at the band of outcomes, okay, for seven day click meta optimization, and we look at how much the incrementality factor differs from the platform reported seven day click result, the results are not that wide.

[00:31:46] Taylor Holiday: Meaning how much value there is to be gained from using the benchmark to using your individual precise outcome is actually often very small. Now if you're spending enough money. That might offset that reality. It might be true that that [00:32:00] still is the best area to test. Alternatively, take something like non-brand categorical search or branded search or app loving, where right now we see pretty wide error bars on incrementality relative to the platform reported result.

[00:32:15] Taylor Holiday: That could fundamentally change how you view the channel. The increased level of precision there really, really matters. And so we, we sort of, we talk a lot about this idea of like, how precise is our truth or how good is the current truth we're operating on. And where can we improve it to make the largest business impact?

[00:32:33] Taylor Holiday: And this is really important, where we bring to the table a series of historical test results that inform a decision based on your current spend levels, the expectations of your performance relative to the efficiency expectation, and how much is to be gained if we get new information here. It's really, really powerful then to turn that into a roadmap of decisions that allow us to get better over time, and also in the absence of the individual test result allow us to use a more precise starting point [00:33:00] than just the platform reported result itself.

[00:33:02] Taylor Holiday: So all of this. Allows for a system that allows us to start more accurately together from a historical set of results that we use across the business. And then two, to help you decide where to start your individual testing journey to get more impact faster in your business. 'cause these things don't move quickly.

[00:33:18] Taylor Holiday: So all of these are, again, pieces of a system that is powerful that allows us to provide you something really effective in the context of the service. Okay, Luke, so let's talk about this now. So we've got measurement, we've got channel allocation, we've got budget, we've got good data. Now we have to act day to day.

[00:33:37] Taylor Holiday: Now we have to turn this into media. Buyers have to make decisions about optimization within channel on a day in and day out basis. How are we using the system to provide that structure?

[00:33:50] Luke Austin: Yeah. So the next step here is now you have to take all this information and you have to set what your t oass. Should be in your Google non-brand campaign, right? And you have to set what your mineral ask should [00:34:00] be in your, in your meta campaign. And again, one of the things that we're operating as a sort of a baseline assumption here is that we're media buying under a cost controlled.

[00:34:10] Luke Austin: Structure where we have efficiency constraints that are set at the specific target to the business objective. And then we have budget that is flexible against that opportunity, right? So if there's more opportunity, it's spending more. If it's under the efficiency expectation, it's gonna, it's gonna be holding, holding that back.

[00:34:25] Luke Austin: So under, under that sort of a structure. Of, we have cost control set up across our campaigns. Then the, the next decision that has to be made is what, what should the cost control be set up? What should the mineral be set up? What should the T Ross be set up for? My Meta and Google campaigns, and this is where. How we describe this is that the results of your incrementality test, what they're going to provide you with is an incrementality factor. And the incrementality factor is the connective tissue between the measurement and the connection to the business level and the ad platform optimization. It is the [00:35:00] thing that connects these things together and helps to operationalize them in that way.

[00:35:04] Luke Austin: So the incrementality factor, when you get back the result of an incrementality test, you're going to see. How much incremental revenue was driven for that test against the budget that you had. And it's gonna back into an incremental ROAS expectation, right? The true I oas, the IR OAS can then be compared to the on-platform reported roas for that same channel or tactic for that same time period. you can understand the relationship between these two things, which is the mo, which is the most important, which is we need to understand in relation to what the platform is reporting, what is the true incremental impact of this channel? Is it higher or lower? And what percent specifically is that?

[00:35:42] Luke Austin: And so we talked about this a earlier, briefly we gave the example of meta acquisition and an incrementality factor of 120%. And so that is saying if your test result is 120%, then. You can expect that the relationship between your on platform reported ROAS and the true incremental impact, it's underreporting by that, by that amount, [00:36:00] right?

[00:36:00] Luke Austin: And so the that is then how you back into what the campaign should be set at in terms of the efficiency expectation. You have the business objective that's set and then you're calibrating your adjusting that for the incrementality factor for that specific channel or tactic. is the on platform target that is set because we aren't able to feed it back, you know, optimize against my geo holdout test in IO As result, we have to connect it back to the on platform reported result.

[00:36:26] Luke Austin: We need to draw a relationship between these two things and the incre mentality factor. Is that connective tissue where we're able to get test result relationship to on platform, set the campaign

[00:36:36] Taylor Holiday: I.

[00:36:37] Luke Austin: accordingly, and then allow the budget to be flexible against that outcome. Now understanding as a quick caveat, that as budget levels change dramatically, increase or decrease, these things need to be retested either through a scale test or another holdout test, depending on which direction you're going.

[00:36:54] Luke Austin: And so the testing roadmap never ends in this way. It's not one and done. You tested meta, you know what the relationship is. It changes [00:37:00] throughout different time periods of the year. Seasonality as a budget increase and decrease, but at that point in time, the best available source of truth you have for that channel's contribution is your most recent incrementality test incrementality factor.

[00:37:11] Luke Austin: That's where you're gonna go set your meta and your Google campaign targets.

[00:37:15] Taylor Holiday: So I think what's really important to add here and I, is that I believe it's our job to represent, to meta the in or any ad channel, the information that it doesn't have visibility into, right? So if we think about that, those are things like we talked early on about your cost of goods. Your ability to understand your actual contribution margin.

[00:37:36] Taylor Holiday: And in this case, it's also the incrementality test. Now this is why some it's also can be useful to run things like a CLS or conversion lift study with a brand with meta. In light of this, you want to provide insight into the actual, incremental impact of your channel and the marginal value.

[00:37:50] Taylor Holiday: That's your job. As the media buyer or gross judges, the thing that you have to represent them meta doesn't have is those, those items, they don't have that in their optimization process, and so you [00:38:00] must build a relationship between the thing it's optimizing for. And this is where I'm gonna caveat and say.

[00:38:06] Taylor Holiday: If you aren't opt, let's say you're using North Beam Apex Electro, which I know some people use, you're optimizing for an MTA result measured in north Beam. Well, then you have to build a factor to whatever you're optimizing for, okay? Whatever the system is optimizing against, you have to represent the gap between the systems knowledge and your actual business outcomes.

[00:38:26] Taylor Holiday: Okay? So whatever you're optimizing for, in our case, we're usually optimizing for seven day click. Meta reported revenue, right? That's what the system is actually allocating the budget against. And so we factor into that metric. So I think the key is just to understand that your job as the human still in this world is to represent the information, the the, the computer doesn't know, right?

[00:38:47] Taylor Holiday: And to hold that as the indication and turning that into an IROs allows you then to act with visibility to the performance of your media. Now, Luke, I'm gonna ask you a harder question, and I don't know that we get into [00:39:00] exactly how we think about this. How do you encourage media buyers to interact day to day with the reported IR oas in any channel?

[00:39:17] Luke Austin: So in this system, you should have one target for the business and everything to be standardized against that, against that target, which is. Again, going back to the spending power model, setting total budget allocation, and it's going, you're defining your business objective at that level. Maximize contribution margin, maximize revenue at break even, that's going to get you to what the business level efficiency expectations should be.

[00:39:41] Luke Austin: The return against the dollars invested at that level. is going to then define what that metric is for your business that every, that all your incrementality factors are applied to, right? And so the incrementality factors, you only need them to go and set the platform reported number. But it in stats [00:40:00] we've, what we've built in our home dashboards, we see every metric, every single day across each of the channels.

[00:40:05] Luke Austin: And we built out encompass for our track, our tracker tabs for each of our for each of the ad channels as well. Is we're looking at I oas from each of those channels reported. So it's taking the on platform reported number and then it, and then back adjusting it based on the incrementality factor to give us an I OAS number.

[00:40:20] Luke Austin: And so what we're looking for the business, if the, if the business level expectation for return is, is a two to one on dollars invested, right against that, against in that time period, then we're looking at a two to one on each of the channels after it's been adjusted for incrementality. And so Google brand campaigns. Could be reporting an eight x ROAS for this point in the month. But in Sta and in our tracker tabs in Compass, if our incrementality factor is 25%, what we're gonna be seeing is a two x against that eight x in platform, in platform number. And that's what we're looking at against e each of the channels.

[00:40:53] Luke Austin: The, it's adjusting in status for the incrementality factor, and that's all going to be aligned with what the, return [00:41:00] expectation is on the business level, so that on a day-to-day basis, basis, we're looking for that two x and anything above that two x we're pushing more dollars into anything below.

[00:41:09] Luke Austin: We're looking for opportunities to, in to increase its efficiency. But the, the system, this system allows for looking at one number, standardizing the reporting across each channel, and then being able to optimize on a daily basis against that after, after it's been adjusted based on the incre mentality factor.

[00:41:26] Taylor Holiday: Okay, so I have my target. I'm looking at it. I've set my targets correctly. I've got my bid caps, my cost caps, my T Ross, whatever I'm using in each platform set appropriately. Now, the real challenge, what happens, I'm two days in, my target's set correctly, but my performance is 30%. Below that expectation, I'm running seven day click optimization.

[00:41:52] Taylor Holiday: What do I do?

[00:41:54] Luke Austin: So. Seven day click optimization seven day window would [00:42:00] be the minimum that you should assess the, assess the outcome for that campaign within right. Google is even wider on a, on a typically 31, 1 or 3 7 1. And so there, you, you should be looking at the delayed robots, but the time, the time window in which you're assessing the efficiency of each of these, of each of these platforms is, is really important to know what adjustments need to be made or not.

[00:42:19] Luke Austin: 'cause if meta is still spending against that campaign, that you have the target set in line with the business objective, then Meta has. Infinitely more signal than you do on seeing potential for it to perform at the expectation over the course of a longer window. So the short answer in that case, in most cases, the answer is do nothing.

[00:42:37] Luke Austin: In that case, you're two days in to the beginning of the month and even to the seven, seven day window. Let it keep running in a longer time window, it's going to get to the expectation or really close to it. And then outside of that, adjustments can be made the sort of like more

[00:42:54] Taylor Holiday: okay.

[00:42:55] Luke Austin: Strategic answer to that would be, I would rather get ahead of volume early on in the [00:43:00] month and then make up efficiency later in the month as well.

[00:43:02] Luke Austin: So I, I would prefer that outcome that I'm pushing more volume. In that scenario, likely my new customer revenue is also ahead of Target 'cause my spend is ahead of target. My contribution margin is probably the thing that's suffering two days in, not that worried into it, like I'm getting ahead of volume.

[00:43:15] Luke Austin: I can make up some contribution margin later in the month, but that's an easier game to play than trying to make up a volume gap with, with two days left at the end.

[00:43:23] Taylor Holiday: Yeah, I think it's really important to understand that because everything is reported on time of purchase. What that means is, let's say you try and scale up your spend today, what you're dealing with is, let's say you've been spending $500 a day for a week, and then all of a sudden you decide you want to go to a thousand, well, you're reaping the benefit.

[00:43:41] Taylor Holiday: Of the previous $500 a day of spend. So think of every day's spend as a cohort of future value. A lot of it's realized on day one, but it's not all realized in day one, right? That cohort is gonna produce value into the future of spend. And so you're stacking all these cohorts historically that are realizing value today.

[00:43:59] Taylor Holiday: [00:44:00] But then all of a sudden I massively increase the spend on the first day of a cohort. So I have a, a fresh new cohort with the highest amount of volume. What that likely means is that it, you should expect to be inefficient in the short period, right? And that you're going to realize that tail value.

[00:44:14] Taylor Holiday: But for a while, that increased spend is launching brand new cohorts of one day spend that are gonna be counterproductive in the reporting of a a seven day click with a time of. A purchase measurement system. This is a really hard thing for people to conceptualize, and it's why media buyers are so prone to pulling back spend against short-term results.

[00:44:34] Taylor Holiday: Versus seeing each day's spend is you have to imagine if you spent a thousand dollars today and then you spent no more money in the future, what would the rule ask of that entire cohort be? If you look back on it in seven days, would it reach your goal? Yes or no? That's the actual measurement system that you're trying to use.

[00:44:50] Taylor Holiday: But instead we're we're scaling spend into these windows where we're looking at it and going, oh, why is this so inefficient when we're trying to stack these various cohorts on [00:45:00] top of each other in different ways? This is why I really wish. That meta went back to time of impression reporting. So this is actually one of the benefits you can get out of sometimes a third party tool because it does really mess with people as they're increasing spend.

[00:45:12] Taylor Holiday: And so it is much easier conceptually to spend a lot in the beginning, stack a bunch of those cohorts, pull back on the spend, realize all the value, get to the impression level that you want, but you just have to understand that the second you pull back on spend, you have some. Effect on the future revenue.

[00:45:28] Taylor Holiday: There's some future realization that's not happening now that spend didn't occur. So still some complexity in, in the data management. The key is to get people out of it on an individual basis as much as possible. Get to wider windows and understand what you can do. So, I hope this helps lay out the pillars for all of you in terms of what we think about in terms of building a modern media management system.

[00:45:51] Taylor Holiday: Operationalizing incrementality. Again, great blog post written by Luke on the website. We're gonna turn this into a PDF that we're gonna distribute for people to [00:46:00] check out, and we would love to walk you through how we could help you bring this to life. There's so many more pieces of the system turning this into a full.

[00:46:07] Taylor Holiday: Fully realized daily view of contribution margin, aligning to your p and l, turning this into a creative expectation through our creative demand model that actually allows us to then build an actual expectation of how many ads we need to make, what kind of ads we need to make, integrating your marketing calendar.

[00:46:22] Taylor Holiday: These are all pieces of what it takes to deliver a system that produces predictable, profitable growth. We spent a lot of time and energy on it. This is a big piece of helping. To update our internal systems. So Luke, any last words you would leave people with as, as they think about taking on this process?

[00:46:38] Luke Austin: We're, we're generally interested in creating the best system for answering each of these questions, and we'll continue to look at across our data set. And like I mentioned earlier there, there's things that we're adjusting right now, and we'll update this blog in probably a month or two with some slight adjustments and improvements to this framework based on what we're seeing.

[00:46:54] Luke Austin: And so follow along with us. We're, we're generally interested in that pursuit and reach out if you, if you see [00:47:00] inputs that are helpful in your process that you see missing from here and big gaps. genuinely interested in creating the best system available and having the best tooling available in, in the space to be these question.

[00:47:10] Luke Austin: So, appreciate you going long.

[00:47:12] Taylor Holiday: Yeah, and like to that point, go read the blog post and then comment on this video on YouTube. If you see this like. Critiques, what are the holes? What would you do to update or improve against this modeling? If you do that, I'll get on a call with you directly and work through your system and our system.

[00:47:29] Taylor Holiday: If you take the time to read Luke's blog and you comment on the YouTube video where you think there's a hole in this process or what you think about it, I'd love to work through that. 'cause again, our interest is genuinely building the best system, and I think we've put some pieces together. Are the best in helping to do that.

[00:47:44] Taylor Holiday: But would love to hear your point of view and see if there's things that update within your individual system that might be helpful. Let's have that conversation. So appreciate you listening, taking the time. Read the blog, comment on the YouTube video, like and subscribe. We appreciate you turning it.

[00:48:00]