Measurement Mistakes That are Costing You Millions

[00:00:00] Richard Gaffin: Hey folks. Welcome to another episode of the Ecommerce Playbook Podcast. I'm your host, Richard Gaffin, Director of Digital Product Strategy here at Common Thread Collective. And I'm joined as I always am by our CEO here at CTC, Mr. Taylor Holiday. Taylor, what's going on, man?

[00:00:15] Taylor Holiday: Nothing. I'm just standing at my desk and everything's about to fall off. Trying to figure out what's happening here. But

what's up man? How are you? I'm good.

[00:00:23] Richard Gaffin: Doing fantastic, Taylor. Appreciate it. Clean shaven, fresh look, fresh week.

[00:00:28] Taylor Holiday: Yeah, I was, I, I got a lot of mockery for my beard containing too many gray and white hair, so I just had, I had to get rid of it. Had to go,

[00:00:35] Richard Gaffin: Oh, it looks distinguished Taylor, you know, 

[00:00:38] Taylor Holiday: to be distinguished yet. I'm just not emotionally prepared for that as my, 

[00:00:41] Richard Gaffin: I'm just coming from a place of not being able to grow facial hair at all, so I'm jealous of everybody's, no matter what it looks like.

so I'll just 

[00:00:47] Taylor Holiday: well, you, you're so youthful, Richard. It it's, it wouldn't suit you anyways.

[00:00:51] Richard Gaffin: I appreciate that. All right, well, so today we're talking about this week rather. So at the end of this week, on Friday, we're releasing the latest episode in our hit Sharpen Your sharpen your Skills series on YouTube. This is season two and, I saw the headline of this episode as we were kind of rolling it out or preparing it to be rolled out is called Starving Giants. And the idea here is about this sort of phenomenon that happens to brands in the 50 to 150 million range. So eight to nine figures. And obviously starving Giants is kind of a, a eye-catching headline, let's say. It sounds like. There's sort of a real crisis situation going on. So we were hoping to, on this podcast, kind of break that down a little bit, give you a little bit of a preview of what this episode is gonna contain and give you kind of a sense of, of just the thought behind it.

So Taylor, let's, let's get into this Starving giants, who's starving and why

[00:01:43] Taylor Holiday: Yeah, this is good timing for a couple reasons. 

One, like you said, we have the second episode of the the season coming out, which if you, if you haven't watched yet, go subscribe to our YouTube channel. Episode one is out about creative volume in the system that we have there. Would love your interaction with this series.

We put a lot of energy into try and to make these episodes in particular, thoughtful. And this next one is. Related to a problem that I see affecting in particular brands in the 50 to 150 million range, where they're sort of, pressing up against some growth challenges in this next phase. And it also parlays nicely today.

Andrew Ferris came out with a great podcast with Olivia Corey from house about. The incremental impact of YouTube and these things are actually really connected in a way. And what it has to do with is that I'm watching these larger brands begin to struggle to drive incremental growth in their core paid media channels and their efficiency is sort of degrading and there's a, at, at the heart of many of it, I think is a. a.

measurement system that has served them incredibly well up until this moment, but is failing to serve them in a new era. And it relates to a primary dependence on shorter term measurement, in particular MTA in a single channel being their.com. While their distribution as a business has broadened beyond that and their media mix has, has also broadened beyond primarily click-based.

System. So two things when I, let me just make these even clearer. The distribution for the product has broadened, meaning they've gone from just selling on their website to now also selling on Amazon and also selling in retail. Okay. That's the broadening of distribution that has occurred, and then their media mix has broadened beyond just meta and Google to now include podcasts, YouTube, television, where you have a more complex system of assessment of impact.

On both where the product's being sold and where it's being advertised. That system now demands an evolution of how you assess the performance of those media dollars. And for many brands, the internal legacy systems that they've built to get to that point are actually causing them to view their media as less efficient than it actually is.

[00:03:54] Richard Gaffin: Yeah, so actually let's, let's zero in on that because that seems kinda like the main issue is that although reported efficiency maybe is going down on D two C, the actual efficiency of the ad spend across all distributions. Is actually maybe maintaining or going up or, I dunno what the case is, but rather it's not a crisis in the same way. The idea being that the impact of the paid media dollars or, or the paid media dollars impact all of those other distribution spaces. So let's talk a little bit about then why the, why the current system isn't adequate for that.

[00:04:23] Taylor Holiday: Yeah, so what happens is, is that in and, and look, we are at the center of a lot of this initial system that was built, which is that brands would be measuring their media dollars. I. In using either click-based attribution within platform, we were a big proponent of that. Or MTA solutions. There's lots of them out there.

I don't need to name any of them individually. And the primary sort of incremental or hierarchal measure above that was a MER new customer acquisition efficiency. So you would take your new customer revenue on your website divided by your total ad spend equals the efficiency of your new customer acquisition.

That system of measurement, something we have become a proponent of. Was a highly effective system for measuring e-commerce growth in what I'll call phase one, which is all advertising exists to drive growth in your.com channel. Okay? That system of measurement supports that highly effective, and what happens is brands build infrastructure around that.

So all their internal reporting, everything they look at is NC ROAS or A MER every day. And so that's how they sort of begin to ingrain the idea of assessing the performance of their media dollars. But what happens is two things as you mature that all of a sudden render that system actually not just in insufficient, but actually problematic in assessing your dollars.

Okay? So one is that as you broaden distribution, so let's say usually the first step of this is now you roll your product out on Amazon, and in many cases the merchandise is actually duplicitous. Meaning you're selling the same things on Amazon that you are on.com. Okay. Well, the idea is that channel is supposed to be incremental revenue growth, and there's certainly incremental revenue growth that happens as you expand distribution on the Amazon, but, but there's also demand capture that happens now on Amazon that would've happened on your website.

In other words, when you provide the best shopping experience in the world to people where they can get same day fulfillment of a product that functionally offers them a cheaper way to buy it because they're probably prime members and so they can get free shipping. On Amazon that they can't get on your website and they can probably get it in two days that they can't get it on your website.

The demand you create at the top of the funnel on your media dollars, wherever channel that is, is now going to be satiated in two different places. So if you don't. Begin to assess the measurement impact or if you don't begin to measure the impact of demand in both places with the same ad dollar. So the same Facebook ad is now creating demand that's being realized in two places, just that one alone.

And this is the first part of the maturity problem, is that if you don't measure that, you will begin to see a degradation of your NC ROS or a MER on your website, even though. What you're really doing is driving broader impact on Amazon. And what I see inside of organizations is generally these things are wildly split apart.

They're different p and ls, different silos run by different people, and this one problem alone will kill you. It will kill you. You cannot take your product and make it available on an amazing place to purchase. It's gonna compete directly with your website. And then expect your ad dollars to be as efficient in your.com channel as they would be before that distribution change.

[00:07:22] Richard Gaffin: Right. So let's, well, I, I just wanna take a step back. And, and previously you'd mentioned that this, that this is a issue of this particular moment. Right? Something that you're sort of seeing more and more. Is that a, a sort of a result of just. Certain D two C brands that we're familiar with growing into the eight nine figure range, or is there something specific about this time, 2025, whatever, where this sort of issue has become more prevalent or noticeable than before?

[00:07:48] Taylor Holiday: Well, so, so you're, you're dealing with all these like, compounding factors of efficiency de degrading. So like, one is just like category competition growth. So if you take, you know, some of, think about some of the most highly competitive categories in e-commerce and think about you have now, like functionally, if you just think about the efficiency of your.com revenue.

A couple of things are happening is one, there's more competition in the category generally than ever before. So if you're selling, you know, t-shirts, right? As an example, there are more competitors to sell t-shirts on the internet than there ever were before, just on that alone. Now you functionally created a competitor to yourself in Amazon, like where if all of your media dollars are expected to drive.com, demand.

What's what's just happening is that as you mature, as the category matures, as the competition increases, there just becomes more competition and you are further out on the customer acquisition curve already. So you're already at a place where you are fighting for efficiency. Just based on the maturity of the customer curve, right?

So you're further out on the customer curve, it becomes harder to acquire the next chance of customers efficiently. So you're, you're fighting your own marginal frontier. And then as you increase the, the competition from other places, so you get things like your categorical search, degrading inefficiency and things like that you get brand conquesting between competitors that drives the inefficiency there.

Now you functionally create a competitor to yourself in that same channel. And so all of that contributes to a part of the competitive. Category maturity for E-commerce that leads to these things suddenly appearing as if these ad dollars are getting worse and worse and worse, and that this is where the danger is.

You pull back because you're trying to get to the A MER efficiency on your website, and all of a sudden the whole thing begins to slow down. 

[00:09:28] Richard Gaffin: Right. So I think that's, that's a good point that you just made. That they appear to be, the ad dollars appear to be inefficient. Which I think sort of then what we could say is that these starving giants, maybe the issue is fundamentally that they actually aren't starving, even though they feel that they are

and are pulling back and actually the issue, they're starving themselves when

they weren't actually 

[00:09:46] Taylor Holiday: Because their, their measurement system doesn't account for Amazon.

and, and and like, it, it, it, looking at NC Ross or a MER if, for a brand like this is just fundamentally the wrong measurement system. You are, you will not, you will not appropriately assess the impact of your advertising if you do that.

So that's like problem one. And then, sort of problem two is like more generally if you don't have, so the way to get to that answer right is to, to be able to do an incrementality study, a holdout on the impact of your advertising, whether that's Facebook search whether that's YouTube, tv, whatever, whatever the core dollars are.

Ideally you have, have this result for all of 'em, and you assess the impact in both places so you, so you can get to a view of the total impact of that ad dollar. Right. Because it might be a dollar 20 on.com and then it might be another 60 cents on Amazon, so it's a dollar 80, which is the difference between not profitable or profitable.

Right. It, it is the fundamental distinction between where you would scale or pull back. And that's what I'm seeing is that if they, if you assess the measurement on.com only the signal is this is not profitable pull back.

But if you include Amazon, the signal changes to this is positive scale and. That single indicator alone will fundamentally alter gas break dynamics in, in your growth.

So, so as you expand in that channel, you have to expand the corresponding measurement system. Okay.

That, that's, that's problem one.

[00:11:07] Richard Gaffin: Well, so I, I, okay. Actually that's probably one. Let's go onto problem two. 'cause I was gonna say like, how do you, is it as simple as just doing a holdout test and, and that's sort of solves your problem? Or, or how does it, how does it become more complex than that, I guess?

[00:11:18] Taylor Holiday: So the other thing that happens is that. No matter how hard you try to create perfect exclusions, the bigger your brand gets, the more customers that you have that are in the category of existing customers, the more of your ad dollars that will be spent onto existing customers versus new customers. So when you start to measure everything on NC ROAS and A MER, when a growing percentage of your money is being spent on driving demand from your existing customer base.

You also develop an inappropriate system of measure. 'cause those ad dollars aren't 

even attempting to generate new customer acquisition efficiency. And so, brands need to begin to have a point of view on the incremental impact of both new and returning customers and have an expectation of the efficiency by which they acquire existing customers.

And so really what this is, is that I think in e-commerce. We have this really flawed idea of of existing customers. And I, I've contributed to this, and I think this is an evolution in maturity of the business, which is to say that a lapsed customer, someone who hasn't bought from you in years, is not your customer.

There's no indication and there's no data evidence that they are coming back on their own and you have to advertise to them. And so some of these brands that are in the a hundred, $150 million range, they have, they have millions of customers and they're literally hard excluding them from all of their ads.

It's like these people have not bought from you in years, and all of the data says that they are not coming back, and yet you are excluding them from all of your advertising. 

And so there begins to need to be this balance between also the way in which the dollars are allocated and so that we need to actually begin to siphon the measurement system to say, okay, these dollars actually went to new customer acquisition, and so we have to get a view of the new customer acquisition efficiency between my.com and Amazon with the ad dollars that are spent on that purpose.

Then we have to have a separate view of the, the dollars that are spent on driving. We would call them like reactivation of laxed customers. Now I, again, I think we should be precious about holding out. The active customer state, so customers that are likely to purchase, but in reality, this is probably like a six to eight month cohort for most, most brands, 

where that's as tight as the exclusionary windows really should be.

Because after that, all the data would say that we're not, those people are not coming back. You 

cannot advertise to them as if they're going to come back. Now that that exact window is different for every brand and should be looked at individually. But the point is just that this idea that a customer is yours forever when you're in year 10 of a brand and somebody hasn't bought in nine years.

Stop holding them out, bring them back into the funnel.

[00:13:49] Richard Gaffin: Gotcha. So it's, it's less an issue of, of that bringing returning customers back into the fold is an issue as much as it is redefining what a returning customer is and treating that as essentially acquisition of some kind, I

[00:14:02] Taylor Holiday: That's right. And, and so e especially as you get to, and the reason I, I say that this problem becomes more pervasive as you get to television and app loving and, and, YouTube is because identity resolution between both channels becomes harder, right? So if you 

think about this idea that like One App Loving As example has no identity resolution, so you, you actually have no ability to control the allocation of your dollars versus new return customers.

So 

that's like a unique challenge. Then you think about something like trying to create a customer identity resolution between amazon and.com. Almost impossible to do, 

right? 

[00:14:34] Richard Gaffin: quick, define identity resolution.

[00:14:36] Taylor Holiday: So the ability to determine if this customer has purchased from you before. 

So if you think about you have a customer ID in your CDP or your database and you are you looking for identifying attributes that would tell you, oh, this person is that previous user that, that they are a newer existing customer and I can confidently resolve that this purchase is attributed to this previous customer and I can 

map their history.

Together, right? There's all sorts of challenges between doing that on Amazon and.com. Then you layer in retail and it becomes very difficult to actually create clear identity resolution between no new and returning customers. So that's where what you're really looking for at a certain point is incremental dollars from any customer type.

I think that there, there's a lot of brands that have grown up on this, this new customer acquisition engine in.com only that are so still so committed to this ideal that until they expend beyond that, they're just gonna continue to suffocate their own growth. And so, and then like the, the, the measurement even just becomes disingenuous where you'll just take like a channel like app 11 or all of your meta dollars and you'll just pile them all into the, the, NC ROS or A MER calculation and it's just like, whoa, that's.

If those dollars are not actually all going to new customer acquisition, and if you were to isolate it out, it'd actually be much more efficient. And so I think that's where you've gotta think about what, what is the actual goal of both? And how do we make sure that we're actually appropriately considering how efficiently we're acquiring customers or not.

[00:15:58] Richard Gaffin: Yeah. So in, I mean, it strikes me that like the, the measurement conundrum that this brings up is feels kind of unresolvable or it feels like, it's like we're sort of moving towards sort of old school, I dunno, the 1960s thing of like, well, the top line went up, therefore the advertising must be working.

Like, that's kind

[00:16:16] Taylor Holiday: No, no, no. Well, So I would just say that what we're, what in, in these cases, when I think about incrementality studies.

What I, what I would encourage you to do is that like, let's, let's go to the YouTube example now that, that Andrew and Olivia were talking about on their podcast today, where across 191.

Incrementality studies done by house on YouTube, they found that there was an incrementality factor versus platform reported results of 3.76. So what does that mean? That means that if, if Google or YouTube was reporting a one-to-one roas, that the actual average incremental factor on that was actually a 3.76.

So it'd be like you're getting $3.76 for every dollar spent. But this is important. This is not a distinction about new customer acquisition efficiency. This is just saying incremental dollars. In total, 

there's no, now you can do it. You can run the distinction between new and returning, but the question is, who cares?

Like if I spend an ad dollar and I get $2 and 80 cents from existing customers and 80 cents from new customers, the question is, how would I think about that next dollar? Now, you may hypothesize that that would create a scale limitation and you would need to do a scale in test to understand how much that incrementality holds.

If the weighted return was more to returning your new customers. But the point is that if you just try to isolate the return of new customers, it wouldn't be $3 and 76 cents. It would actually be lower than that. And that's where I think for a lot of these brands in these channels where identity resolution is hard and where you're trying to assess the impact between amazon and.com, you're gonna have to let go a little bit.

Of this 

idea that new customer acquisition is the only thing that we care about, especially when you have millions and millions of historical customers that haven't brought from you in a long time. So there's a size consideration there as well.

[00:17:58] Richard Gaffin: Yeah. Okay. So, so then I, I wanted to talk a little bit about 'cause we, we'd mentioned this before we hit record. Is that part of the. What we're trying to assess here as well is that, and part of the reason that pulling back on, let's say meta dollars or whatever be has such an impact or becomes this sort of starvation thing, is because there's the like, sorry, meta advertising is the way you kind of exist out in the world.

Like that is the way that you're sort of announcing your existence. Therefore, meta is having an impact on those downstream distribution channels like Amazon. Like retail, whatever. And so there is some connection there as well. Like what? How do you consider that in terms of measurement and efficiency, that sort

[00:18:40] Taylor Holiday: Yeah, it just becomes another place where this problem compounds, right, is that even if there is an incremental, fractional, tiny value in your retail landscape, it's just unaccounted for right now. What's happening is a lot of times all the ad dollars are showing up on the p and l of a single channel, but they're driving impact everywhere.

And if you don't get visibility into that impact, you are going to reduce how much you would invest in advertising. And the, this is just what I see is that I see brands that they're just spending less than they should in their growth is plateauing because they just don't have a way to account for its impact.

Now, again, I'm not just saying that you should just, the alternative here I've also seen go bad is what you just assume impact. You just say, well. There's 40 cents of impact here and another 20 cents there, and you just go, it's even better than we think we should. Keep going. No, no, no. You should obligate yourself to getting evidence of impact and that's what good holdout studies do.

But you cannot broaden your sales distribution and not come up with some understanding that and add on YouTube and add on meta. An add on TV is going to drive demand in everywhere you sell the product from. And, and when I, and some of the data I've seen is that like the impact, especially for some of the more top of funnel view based platforms, television and YouTube, more of the demand can be captured in Amazon and retail than on.com because it's not a click base, it's not a clear linear path to purchase.

So. Especially when you go there, so, so for brands that have broad distribution across retail and YouTube or and Amazon and.com and are running television and YouTube, you absolutely have to account for this. Otherwise, you will be under rating the impact of the advertising that you're doing.

[00:20:19] Richard Gaffin: Yeah. Okay, so let's I'll ask a question I like to ask when we have these types of discussions, which is if you are in this situation. As many of our listeners probably are. What, what's the one thing that you can do right now to begin to mitigate this issue?

[00:20:33] Taylor Holiday: So I would, I would, I would examine your internal reporting structure. Okay. So I like to say that your system is perfectly designed for the outcome that it's getting. Okay? And I would ask you every day, when you look at the performance of your media dollars, what metrics do you look at? Okay? And what I, what I bet is that most of them look at.com revenue.

MER or, or ACOs or some people call it blended roas, whatever it is. I bet they look at that and then they look at some efficiency of new customer acquisition. And I would ask what dollars are in there and what revenue is in there in that calculation? And if that's the case and Amazon is not part of that, but yet you are on Amazon, you have a problem.

You have a problem. So first is the identification of the problem. Two is if you, if you're in retail in particular, your owned retail or other online retail marketplaces, or you're selling on nordstrom.com or whatever, like there, there is, and that revenue's not accounted for and included in the way that you look at your daily impact of advertising, you have a problem.

So that's first is that examine your internal systems of reporting. What revenue do you look at every day? What what a MER or new customer ROAS composition are you looking at every day? And then how how is that comprised? Okay. Once you identify the problem, now you can begin to say, okay, what is the step to creating a solution?

So, the step to creating a solution is creating visibility first, before you even go do a holdout study, is just get your Amazon and your Amazon revenue and your.com revenue viewed in the same place every day against the ad spend. So get a total MER, get a total a MER across all of it. Okay? Now we can at least see visibility into that and look at contribution margin as well combined.

Obviously, they have different unit economics, things like that, that adds some complexity to that. So get, get it all looked at in one place. This is something we're working on in Atlas. We're very, we have an integration now with Amazon to be able to provide people that daily view of all of their revenue where possible.

Now let's think about our largest ad spend. So probably meta and let's begin with setting up a holdout study to assess the incremental impact of both.com revenue new and returning ideally, and Amazon revenue New and returning. Get that report first. So that's like the, the steps. Check your internal systems for the problem.

Try to combine the reporting issue to unify these things. I would also, usually also in there is like incentive problems. There's a p and l leader of Amazon that's different than the p and l leader of.com. They don't actually, one of 'em doesn't want the ad dollars to show up on their p and l. So this is a problem I see all the time.

It's just literally an internal incentive structure problem or it's a completely different team that's in control of Amazon. There's a different agency. There's like, it's just like they might as well be different universes. So go after that, examine that, unify that. And then I think the holdout study becomes next, and now that becomes the mechanism by which you report all of your efficiency.

This is so like people run these studies, but then they don't operationalize them. It's like, oh yeah, I know that Amazon has an impact, but like we just don't report it. It's like, what? Like what was the point? Like, you know, there's an incremental impact. Yet we're gonna go back to looking at. The MTA every day, it's like, I, I don't, what, what are we doing?

So I think that that's the, you, you then have to operate once you fourth step is operationalize the results once you get it, to actually make your decisions based on that. And then reassess constantly. 'cause it's a static point in time and it's, makes it challenging.

[00:23:45] Richard Gaffin: Yeah, so I mean, it sounds like kind of in summary, it falls into the same category of advice that we give often, like with, with say, our Bridges series, which is that you just have to connect the dots. This is about at least partially being an issue of p and ls being siloed, not happening between parts of the organization, finding any places where that breakdown is happening and resolving it

[00:24:06] Taylor Holiday: Yeah, it's all a system design problem. It is, it is not usually just an advertising problem. It is a primarily a system design, which includes measurement. And step five, really the first thing you should do is watch the video on Friday. 'cause I 

break this down in a, in a way that I think will be helpful to illustrate the problem that you can consider.

Does this apply to us? Is this relevant for the problems that we're facing? And of course, if you need help, we're, that's what we do. We're here to help. Create these problems. And one of our big focuses is the integration of.com and Amazon in a reporting, in a modeling. This is another thing, modeling.

So modeling the 

expectation of efficiency into the future. So those are big things. We're working to update our A MER models to include Amazon revenue, to update our return customer cohorts, to include Amazon, to update our, our all of the, the forecasting that we do to include both channels and then daily reporting on both so that you can see the total impact of your business and make sure that you're investing in the amount of growth that would get you where you need to go.

[00:24:56] Richard Gaffin: That's all right, folks. All right. Yeah, as Taylor mentioned, that video's coming out on Friday the seventh here, so look out for that on your YouTube channel. And of course again, as Taylor mentioned, if you wanna chat more about how we can build these models for you and help kind of. Kind of, build these communication structures for your organization. Check us out common thread code.com. Hit that higher us button. We would love to chat. All right folks, thanks again for joining us, and we'll see you next week. Bye.