We Ran Incrementality Tests … Meta’s Tool Didn’t Win

[00:00:00] Richard Gaffin: Hey folks. Welcome to the Ecommerce Playbook Podcast. I'm your host, Richard Gaffin, the Director of Digital Product Strategy here at Common Thread, and I'm joined for a third conversation about incrementality here by Mr. Tony Chopp, the Chopper, our VP of Paid Media here at Common Thread. Tony, what's going on, man?

[00:00:17] Tony Chopp: Well, yeah, I mean, this is our third conversation about this specific product,

[00:00:21] Richard Gaffin: Yeah.

[00:00:21] Tony Chopp: I feel like almost no 100% of our conversations are about incrementality in

[00:00:27] Richard Gaffin: That's a good point. That's right. Yeah. It's been literally dozens at this point, I'm sure.

[00:00:30] Tony Chopp: Yeah,

[00:00:31] Richard Gaffin: Um,

[00:00:31] Tony Chopp: though. How are you?

[00:00:32] Richard Gaffin: yeah. Doing all, doing all right man. It was good to, it was good to see you all last week. Um. So I think like what we wanna talk about here then is, like you were saying, this is about Meta's incremental attribution tool, the product they've been rolling out.

Uh, just to recap for you guys, in our first couple of conversations, the first conversations was we were about to kick off a test,

[00:00:52] Tony Chopp: Yep.

[00:00:52] Richard Gaffin: uh, which as I understand it, is essentially a meta conversion lift study. On the incremental attribution tool versus let's say seven day click and one debut or whatever the other attribution settings are.

Uh, our second kind of conversation, we checked in on that, uh, test a little bit. Um, saw some initial pretty promising results. Uh, this third conversation. We sort of, my understanding anyway, like completed those tests and actually introduced ano another one, which we'll talk about in a second. And the results, uh, are interesting, let's say.

So without further ado and without spoiling anything, let's jump right into it, Tony, and talk about kind of like since the last time we talked, how, what has developed with this tool.

If you've been paying attention to the news, you've probably seen that costs are going up everywhere and consumer expectations are similarly going up. For fast-growing e-commerce brands, that means increased fulfillment fees, parcel costs, and squeezed margins. With the changes to global trade, brands are seeing a massive spike of 15 to 35%, and with the changes to USPS parcel select delivery costs could balloon well beyond 25%.

In order for your brand to survive, you will need a scalable commerce enablement provider who can adapt to these changing markets and help you deliver the best consumer experience at scale. That's precisely what store does every day for their partners. In 2024 stored saved $130 million on parcel fees for its customers.

And after powering nearly 1% of all US online, black Friday and Cyber Monday sales, they have the infrastructure and integrated technology to help you grow in 2025 and beyond. No matter how many disruptions occur, don't just take my word for it, you can experience it for yourself. They've just rolled out an in-depth mystery shopping report.

This report compares your brand against two competitors and presents you with real. Actionable insights based on industry benchmarks. To request your free report, go to stored link slash mystery. That's S-T-O-R-D link slash mystery. Take the mystery out of your consumer experience with stored.

[00:03:00] Tony Chopp: Yeah. Yeah. No, that was a perfect setup. So you're exactly right. The, the last time we spoke, we, we were getting into the results that we saw from the several client projects that we ran the, the product on. And then subsequently we've run a test with CTCs incrementality testing tool. Okay. And I wanna call out the difference here because this is really important to me. When we run conversion lift studies in Meta or Google for that matter, the critique of those is they are grading their own homework.

[00:03:38] Richard Gaffin: Right.

[00:03:39] Tony Chopp: platform reported revenue to measure the impact. It's fine, I'm not saying it's not useful. I'm

[00:03:47] Richard Gaffin: Mm-hmm.

[00:03:47] Tony Chopp: that's just the critique of it. This third test that we've run. Using incrementality tool is what what we consider to be the sort of highest bar, which is a geo holdout, and the measurement is Shopify revenue

[00:04:06] Richard Gaffin: Mm-hmm.

[00:04:06] Tony Chopp: Shopify revenue. So really important distinction, and to your point, the results that we've gotten back from this third test. Are actually pointed in the other direction.

So what, what we're seeing is there was, um, more of an impact on returning customers, less of an impact on new customers for the incremental attribution arm, but both of them combined were less incremental than BAU, which in this case was seven day click. Okay,

[00:04:46] Richard Gaffin: Interesting.

[00:04:48] Tony Chopp: You and I were talking about this, you know, before we got started here, and what I, I feel like I want to express to anybody listening here is something I, I assume everybody can empathize with, which is, this stuff is tricky.

It's a, it's a journey, not a destination. And what we're after at any point in this journey. Is attempting to increase our confidence level brick by brick, by brick, by brick, we can ultimately get to a high level of certainty around deploying media in any one tactic or another.

[00:05:25] Richard Gaffin: Mm-hmm. Right. Um, and so I, I mean, I think like one, one thing that we were talking about too is, and, and like you're kind of alluding to here, is that it's not. Like this is not necessarily a knock on the concept of incrementality or even the idea that this tool will not become important maybe even in the near future.

It's just like at this point. I, I mean, maybe one thing to say too is like, it, it's not, doesn't necessarily come as an enormous surprise that med is, self-reporting on the efficacy of its tools is probably exaggerated. That's the kind of thing that almost always happens. So part of what we're doing here is kind of holding that to account with our own geo holdout test.

And the fact that the results are a little bit, are, are mixed, I guess is, is again, also not a surprise. Um, but again, that's, uh, the idea there. That's, it's not a knock against. Incrementality being the gold standard or, or the thing that you ought to strive for. It's just more that this tool isn't quite where we want it to be yet.

Does that sound right?

[00:06:24] Tony Chopp: Sort of, I would, I would, I wanna throw some caveats in there.

[00:06:28] Richard Gaffin: Sure.

[00:06:29] Tony Chopp: Um. The, the process of going through an incrementality test arrive at an incrementality factor

[00:06:40] Richard Gaffin: Mm-hmm.

[00:06:41] Tony Chopp: is all in pursuit of setting a target in the platform.

[00:06:45] Richard Gaffin: Right.

[00:06:46] Tony Chopp: so the answer is not

[00:06:50] Richard Gaffin: Mm-hmm.

[00:06:52] Tony Chopp: or one is bad. In this case where we saw. In the test, we, we ran, we saw a higher incrementality factor for seven day click

[00:07:04] Richard Gaffin: Mm-hmm.

[00:07:04] Tony Chopp: incremental attribution.

It doesn't mean one is good or bad. It means that if we were to continue to scale into this product, we would just need to set a higher target.

[00:07:16] Richard Gaffin: Hmm.

[00:07:17] Tony Chopp: That's

[00:07:17] Richard Gaffin: Interesting. Okay.

[00:07:19] Tony Chopp: So

[00:07:20] Richard Gaffin: So

[00:07:21] Tony Chopp: I, I

[00:07:21] Richard Gaffin: interesting.

[00:07:22] Tony Chopp: I like, I like to hammer this point

[00:07:24] Richard Gaffin: Yeah.

[00:07:25] Tony Chopp: That that is the essence of the incrementality testing thing.

It's not like good, bad. It's how do we get to setting a aro target or a cost cap target for a campaign that we believe will create the financial outcome that we're seeking.

[00:07:39] Richard Gaffin: Interesting. Okay, so, so then the question of a a, like, so, okay, so what you're saying is not necessarily that one BAU is more effective than this incrementality tool. Okay. I understand that. Um, merely that the incrementality or the IOS metric or whatever that is reporting as, um. It's just, it just, the report is coming back that the spend is less incremental or whatever than the spend on BAO.

Now maybe the, the better like rubric to judge this by is accuracy, right? So is the idea that, do you have a sense of our numbers with geo holdout study via Shopify rev numbers as opposed to meta's numbers on the, um, on their test or whatever. Um. Or rather, sorry, maybe I should say the results of our geo holdout test.

Do they indicate something about the accuracy of the reporting in meta on the IROS metric? Um, or is there any value judgment that you can make between the two of them based on the results that we've seen? Sure.

[00:08:43] Tony Chopp: sort of, I'm, I'm gonna answer the question, um, with my like spidey sense about. The, the optimization tactic, which I think is supported by the data that we see. Um, my spidey sense is that, um, the incremental attribution product is going to skew towards showing advertisements to warmer audiences. So, and we do a bunch of work to exclude. Existing customers and pixel and everything else. But we know that even in spite of all those exclusions, um, there's a fair amount of existing customers that are still able to through the ads.

[00:09:34] Richard Gaffin: Right.

[00:09:34] Tony Chopp: we can debate like how much or we, we've seen different things. But anyway, the story I'm telling myself is the incremental attribution product is going to lean that direction now.

[00:09:43] Richard Gaffin: Mm-hmm.

[00:09:43] Tony Chopp: think that's o probably an okay thing.

[00:09:47] Richard Gaffin: Yeah.

Get mystery shopped for your brand and two competitors of your choice for free. Stored will provide a detailed report that outlines the specific areas you're outperforming your competitors, and where your competitors are outperforming you. Learn how your consumers truly experience your brand today.

Request your free report at stored link slash mystery. That's S-T-O-R-D link slash mystery.

[00:10:14] Tony Chopp: So in the, the reason the supporting evidence for this is in our incrementality tests, we saw a stronger higher impact on existing customers as opposed to new customers. Okay, so let me put this in really simple terms. When we, when we run a. An acquisition campaign, seven day click the exclusions we can, we can add in, we have an incrementality factor of, of 120%. So generally we're saying that is under reporting of impact. When we run a retention, like a existing audience focus campaign, we treat those as 60% incremental as our, as our default. And this incremental attribution campaign is reporting and showing up more like a retention campaign would.

[00:11:18] Richard Gaffin: Okay. Interesting.

[00:11:19] Tony Chopp: So yeah, that's

[00:11:22] Richard Gaffin: Okay,

[00:11:23] Tony Chopp: far.

[00:11:24] Richard Gaffin: interesting. All right, so in some ways then, I guess it's like it's just generally too early to draw much of a conclusion about what's really going on here. Um.

[00:11:35] Tony Chopp: Well,

[00:11:36] Richard Gaffin: But if there's anything, like, if any conclusion can be drawn, like where, where is your head at right now?

[00:11:46] Tony Chopp: I, the conclusion at right now is we're maintaining a conservative, Curious posture.

[00:11:55] Richard Gaffin: Mm-hmm.

[00:11:56] Tony Chopp: So what does that, that means that if I'm, if I'm running a business right now, and in fact for how we're advising our clients, uh, I'm not gonna move a hundred percent of my media spend on meta into this incremental attribution product Um, but am absolutely going to continue to push five or 10% into. Incrementality tests to understand how this product works. Um, and I want to, I want to sort of add a little bit of color to that, which is, let's say, let's start with number, number one. This product is new. It's brand new. Number two, it's a fundamentally different way of measuring as opposed to click-based attribution.

It's

[00:12:50] Richard Gaffin: Right.

[00:12:51] Tony Chopp: running these. Hold out tests like underneath the hood continuously, fundamentally different. Number three, I believe it's right actually. I think it's the right way to do it.

[00:13:01] Richard Gaffin: Yeah.

[00:13:01] Tony Chopp: even if it's imperfect at, at this time. Uh, and number four, like we just have, we have such a higher level of fidelity on understanding the, the traditional seven day click, full exclusion acquisition setup, um, that. Um, you know, on that, on that confidence interval, we're like, if here's 0%, here's a hundred percent confident, which never exists, but we're, we're way further over here on this one than we are in the incremental attribution. But I, we're gonna keep going side note too. We had, uh, there, a, a house report that came out this week, A big meta incrementality study. Um. Which, uh, I'm looking forward to getting my hands on about 615, uh, 600 or so clients. They, they ran it on and I believe I heard anecdotally that they had some subset analysis on this, this product specifically in there. So I'm really curious to see what they're, what they're finding. But another thing on that note, just to, to bump on that,

[00:14:12] Richard Gaffin: Mm-hmm.

[00:14:13] Tony Chopp: uh, there I've heard through the grapevine, I haven't read the whole study yet, but, um, there. Incrementality percentage on meta click only attribution came back about 115% incremental.

[00:14:26] Richard Gaffin: Hmm.

[00:14:27] Tony Chopp: we've been using 120, uh, for the last year or so, and

[00:14:30] Richard Gaffin: Yeah.

[00:14:31] Tony Chopp: to see like how, how, how close

[00:14:34] Richard Gaffin: Yeah. Close to it.

[00:14:34] Tony Chopp: with their study.

[00:14:36] Richard Gaffin: Yeah. Um, no, that's interesting. Well, like when, when you get your hands on that, we'll do a, a report on the report because I think that would be fascinating to. Compare and contrast with what we've been experiencing over the last year or so. So I think like, just to summarize a little bit of what you're saying is that regardless of the kinda result maybe of this most recent test that we've run, the mechanic behind the incrementality attribution setting seems like the right mechanic.

Ultimately this idea of the 24 7 geo holdout thing. Now it may be imperfect at this point. But at some point when they work it out, this will be the way that we want to do things. And it's just a matter of kind of like going on a journey with it, I guess. Um, as we evolve, as kind of our under understanding of how do, how do you even kind of work with this attribution setting?

'cause as you're saying, it's so fundamentally, mechanically different from what we've been doing before.

[00:15:27] Tony Chopp: Yeah,

[00:15:28] Richard Gaffin: does that sound fair?

[00:15:29] Tony Chopp: yeah, yeah. Totally. Yeah. Like. It, it aims to, one of these episodes we talked about like geo holdout, incrementality testing is the best we have. But we talked about some of the limitations, right? Like we talked about the fact that a point in time measurement,

[00:15:46] Richard Gaffin: Right.

[00:15:46] Tony Chopp: a snapshot in, uh, whatever March,

[00:15:51] Richard Gaffin: Mm-hmm.

[00:15:52] Tony Chopp: how does that apply to Black Friday, right?

[00:15:55] Richard Gaffin: Yeah. Yeah, totally.

[00:15:56] Tony Chopp: so there's like limitations. S in inherent in the way that that incrementality testing works. And again, I'm not, I'm not going to be the person that gives you black or white sort of like good, bad

[00:16:09] Richard Gaffin: Yeah.

[00:16:10] Tony Chopp: me. Like this is the best, this is the best that we have. And still yet there's these limitations.

[00:16:15] Richard Gaffin: Mm-hmm.

[00:16:15] Tony Chopp: So on the flip side of the coin, the incremental attribution distribution feels like the right way to address some of those limitations. The point in time analysis like challenge. It's the scale challenge, like changes when we scale more so directionally it's right.

[00:16:32] Richard Gaffin: Mm-hmm.

[00:16:33] Tony Chopp: on this side are, it's a new product.

We're still trying to get our hands around it, figure out like how, what our level of confidence is in the measurement. So listen, Richard, you know this game as well as I do, it's a

[00:16:44] Richard Gaffin: Sure.

[00:16:45] Tony Chopp: a destination.

[00:16:46] Richard Gaffin: That's so true. It's so true. Um, yeah, no, and it, and it's, it's going to be difficult to have. I mean, I think just like all the conversations we've had about the actual development of Facebook ads as a product,

[00:17:00] Tony Chopp: Yeah.

[00:17:00] Richard Gaffin: a little bit of a black box and it's always gonna be difficult to kind of point to something and say like, yes, this is 100% what's going on here?

This is right, this is wrong, whatever. Like we have definitely informed guesses based on a lot of experience, but I think with this it's just we have, we know what we know right now and we'll continue to get better at that. One thing I was thinking as we were talking is like potentially the discrepancy.

Between our test and the incrementality tool right now, could that possibly be partially explained by the fact that again, they're running off, I mean it's similar mechanic, but because the incremental ROAS metric or whatever is running 24 7 and we have this sort of like restricted by human, you know, whatever geo holdout thing that we're running at the same time.

Is that one of the things I, I'm just sort of speculating here, but like there are. These sort of, these little factors that we're not considering necessarily that may explain some of the discrepancy right now.

[00:17:54] Tony Chopp: Yeah, I, I, I'm be, if I'm being really candid with you, I don't, I don't think we have f like fully wrapped our head around like all of the inputs that could be causing these things to be different. Right. So like. Yeah, my, my instinct, my spidey sense was like, oh, this, this product is like trying to hit a, a warmer audience, you

[00:18:16] Richard Gaffin: Yeah.

[00:18:16] Tony Chopp: And as a result of that, like sort of behaving or measuring out from an incrementality standpoint, more like a retention campaign, um, I think that's probably just like my of bias and like jadedness around, like how the media platforms sort of work in general.

[00:18:34] Richard Gaffin: Mm-hmm.

[00:18:34] Tony Chopp: of have this bias to go after. Who they're gonna convert AK to people that already know about you, AKA whatever. so yeah, so, but I think what I'm hearing you say is like one side of, one side of the testing arm, the BAU side of the testing arm, we have an incrementality percentage of acts

[00:18:58] Richard Gaffin: Mm-hmm.

[00:18:59] Tony Chopp: and it's, and it's a point in time measurement from some time ago. other side of the testing arm, the incremental attribution product. We have a measurement of why, and it's lower, but it's like continuously always, always on. And is that, is that point in time thing sort of part of the discrepancy? Like I think that's a really

[00:19:19] Richard Gaffin: Mm-hmm. Yeah. But yeah, hard to say. And yeah, it's just, just layman speculation on my part, but I like, these are the types of things that I think as time goes on, we'll have a little better understanding of what the cause of some of those things are. But, um, alright. Any, any final conclusions you want to draw here, Tony?

[00:19:40] Tony Chopp: You want me to, you want me to plant, plant a flag in the,

[00:19:43] Richard Gaffin: No, no, no, no, no, no. Conclu. Just any, maybe any passing,

[00:19:47] Tony Chopp: Here's

[00:19:47] Richard Gaffin: parting thoughts. Okay.

[00:19:49] Tony Chopp: We'll see you for episode four sometime in the future.

[00:19:53] Richard Gaffin: Yeah, that's right. That's a good conclusion. One we thing we can guarantee you is that we will be talking about this again at some point as this develops, but. Cool. Alright, Tony, well I think that's gonna do it for us. Short and sweet. Um, appreciate, uh, your expertise, bringing these sort of like frontline things to us on the pod.

But um, alright folks, thanks again for listening and we will see you all next time. See you.