Meta's New Setting Beat Our Old Campaign by 18%

Tony Chopp: [00:00:00] This is like really a fundamentally different mechanism by which the measurement plays out and that paired with the conversion based bidding. Conversion based optimization is like, it, it feels like we're, we're, we're on the verge of something really cool. 

Richard Gaffin: Hey folks. Welcome back to the Ecommerce Playbook Podcast. I'm your host, Richard Gaffin, Director of Digital Product Strategy here at Common Thread Collective. And I'm joined yet again. I think it's been a couple weeks since we've had 'em on, but Tony Chop and McCay Rasmussen are intrepid explorers of incrementality here.

Tony McCay, what's going on guys?

Tony Chopp: Hey, Richard.

McCay Rasmussen: are you?

Richard Gaffin: Great. Love it. So what we're talking about here is actually maybe I'll have Tony, you jump in and kind of remind us what we're talking about. So our team has been testing the new incrementality attribution setting on meta over the last, I guess it's been two weeks now. And we have some learnings, we have some stuff to share with you.

So Tony, quickly remind us kind of, of, of what we've been [00:01:00] doing.

Tony Chopp: Yeah. So, yeah. This is a brand new product from Meta called Incremental Attribution, and it's it's an alternative. Optimization setting as opposed to choosing seven day click one day click seven in one. So it's a, it's a totally different optimization setting and, and fundamentally it's like completely different under the hood in how it accounts for value conversion value from any given campaign.

So it's not, it's not based on any sort of time window. that's like the basic outline of what we're talking about. Now, I don't know if you want me to get into, I, since we spoke last, I've become even more excited about this product. But let

Richard Gaffin: Okay.

Tony Chopp: me pause there.

Richard Gaffin: Well, I, I would say if, if new information has come to light that it's getting you excited, Tony, then I do want to hear about it. So let's if, if it's, if you think it kind of reveals something about particularly the efficacy of this test, then I think it would be definitely irrelevant.

Tony Chopp: Well, let me, okay, [00:02:00] so let me just like lay one thing out, like this feeling that I'm having right now. I, I've been doing. Media for a long time, and I remember

Richard Gaffin: I.

Tony Chopp: let's say 10 years ago before we had conversion based bidding, right? So min ru or cost cap. So back in those days we used to do bidding manually.

We'd bid for like a, a cost per click. And, and I think about like how transformational it was for all of the advertising platforms to give us this outcome based mechanism by which to. To manage our campaigns and now, now we use it a hundred almost all the time. Like it's the, it's the only thing we do. So I, I actually feel like we're in a similar, we're at a similar point right now where this is, this isn't just a little like, of small twist.

This is like really a fundamentally different mechanism by which the measurement plays out and that paired with the conversion based bidding. Conversion based [00:03:00] optimization is like, it, it feels like we're, we're, we're on the verge of something really cool. So,

Richard Gaffin: Okay.

Tony Chopp: but I will take before, before we get into all that specific stuff McCay and I have been talking quite a bit about the, the incrementality measurement and the journey that, that we've been on over the last 18 months. and we've been talking about like quite a few of the. challenges, so like incrementality, measurement, geo holdout. It's the, it's the best. It's the gold standard. It's the best that we have, but it's not without challenges. And I'd love for McCay, if I could kick it over to you to just sort of run through what are some of the challenges that we see when, when we're going through this process.

McCay Rasmussen: Yeah, absolutely. I think. To your point, it's the best we have as of now until incremental attribution came out. But some of the challenges with, with running incrementality tests in, in those benchmarks is a it's point of time based. And, and that can be problematic in the sense that brands obviously go through different [00:04:00] seasons and those different seasons may. And what we've experienced has made change that incrementality benchmark. What we're having to do is run quarterly tasks to, to get kind of this benchmark of, of what kind of lift does our, does our meta spend or does our media spend actually give us in addition to that, even changing spend threshold, so.

Going from, and, and this is kind of in line with seasonality, but going from, let's say spending at this level versus spending at this level, those may have different benchmarks as well. And then just to throw in even more complexity into this is, is new in existing customer mix. You have attribution windows, which again. Historically we've seen seven day click versus seven day click with view have different incrementality reads, and then there's, there's bidding value versus bidding volume campaign type. So there's all these variables when, when you're running these tests. And, and I think the reason Tony and I are so excited about this is this effectively puts an end to all of this.

This is an always on tool where you no longer have to account for all these different variables and, and seasonality and point of time and, and spend [00:05:00] thresholds. And it's just like. Do this all the time, every impression. So

Richard Gaffin: Right. Yeah. So I think like, to, to summarize, like, yeah, the thing about this tool that's so incredible is it's essentially running an incrementality test all, all day every day, no matter what time of day, no matter what's happening, the, this incrementality testing is running all the time. So you have a much clearer sense of the actual performance of your ads.

So let's talk a little bit then about. Given those challenges with incrementality or, or rather, maybe I should phrase, phrase it this way, what exactly have we been testing then over the last couple of weeks? So we've been testing, as I understand it, this our sort of geo holdout our old system versus, and, and kind of like comparing it to the data that we're getting from this sort of I OAS measurement in meta.

Is that correct? And could you unpack that a little more?

McCay Rasmussen: Yeah, absolutely. And that's exactly what we tested. So two cell tests. Both cells had the same same ads. [00:06:00] Same everything. The only difference was the attribution setting. So on one side we had standard attribution click only seven day click versus cell B was incremental attribution. And that was the only, that was the only difference between the two. And yeah. Tony, do you wanna dive into to what we found.

Tony Chopp: Yeah, for sure. So I'm gonna share a slide, which you guys can, you guys can publish.

okay, Richard, can you see my screen?

Richard Gaffin: Yes, sir.

Tony Chopp: Okay, so to McCay's point, everything the same, same creative, same everything. In the campaign, the only difference, the split was one running standard attribution. In this case, seven day click if I'm not mistaken, McCay and the other arm running incremental attribution. So, and, and this is the, this is like the punchline. This is like the why should I care about this, or what, what does this do for me and my business? Ultimately the difference between these two campaigns is one. Had, $3,800 [00:07:00] in conversion value. The other had $3,100 in conversion value. So the incremental attribution campaign had $700 or 18% more sales. So when we look at it in meta, they describe it as a, as a ROEs lift, as a lift over BAU. effectively what this is saying is that this optimization tactic is leading to more sales from the same amount of media investment. Now this gets us very, very excited, right? Because I, I think it's easy for us to kind of nerd out in the technology and like the coolness of it, but for, for all intents and purposes, our, our partners, our clients, the businesses we work with, fundamentally what they care about is the outcome. So the reason McCay and I are super excited about this is because we're starting to see. Incrementally better outcomes as a result of using this tactic in the tests that we're running. Okay. I'm curious if you have anything else to add as you've been [00:08:00] watching this really closely.

McCay Rasmussen: Yeah, I think, I think this just represents us being on the cusp of, of something a huge monumental shift within, within all media buying. And, and in the term, and I guess how I'd put it is, is our job is to, to locate, we'll cut that. Our job is to, to maximize our paid media efforts against the business outcomes we're looking for. And I think like if we circle back to, to what we're doing right now as buyers is, you know, adjusting budgets, adjusting cost controls based on, on our incrementality reads going into the exclusions and really just trying to optimize our media to the best effort. This is essentially taking that a step further and is gonna allow us to be. optimized. I mean, I guess the, the most optimized you could get is an always on incremental attribution. And, and so I think we're, we're just on the cusp of something huge.

Richard Gaffin: Mm-hmm. Interesting. So, so talk to me about like the, maybe a little bit more about the setup of this test because, [00:09:00] so actually, can you pull that slide up again, Tony? That'd be helpful.

All right, so looking at this slide here, so what we're seeing is that essentially optimizing for incremental attribution resulted in a performance lift against the optimization for standard attribution. So just to be clear, just ask a couple dumb questions here. This is not showing that incremental attribution is.

Is more accurate in the sense that standard attribution actually produced a similar result, but the incremental attribution is actually reporting it correctly to the tune of being $700 over. What we're actually saying here is that what this test shows is that incremental attribution actually results in something better.

Is that correct, Tony?

Tony Chopp: Yeah, I, I think that's, I think that's a fair way to say it. That, that it actually produced more of an outcome,

Richard Gaffin: Mm-hmm.

Tony Chopp: more sales.

Richard Gaffin: Yeah.

Tony Chopp: I, I think the, we're gonna talk about like the roadmap and the next steps that we're, we're going at here because [00:10:00] one, of the things that, okay, Tony, make this not confusing. Alright. A bunch of different ways to execute these tests, these types of tests. This version one, this first version that we're doing is all happening in the Meta Ads platform. Okay? So it's ab split test in the meta ads platform and the Achilles heel of. Running the tests in the platforms is that all of the revenue is platform reported.

Richard Gaffin: Mm-hmm.

Tony Chopp: your own homework effectively.

Richard Gaffin: Right, right.

Tony Chopp: So now that's okay. It's a good place to start and we encourage everyone to start here actually. But McCay's roadmap that he's gonna talk about on his pathway of having all hi, all of his all the accounts at CTC fully utilizing this across the board is to do a geo holdout.

So

Richard Gaffin: Gotcha. I see.

Tony Chopp: step of this.

Richard Gaffin: Gotcha. Okay.

Tony Chopp: but to answer your question

Richard Gaffin: Hmm.

Tony Chopp: the, with the visibility that we have from this style of [00:11:00] test, one produced $700 more than the other one.

Richard Gaffin: Gotcha. I see what you're saying. Okay, so, so that makes sense. So then, then just to clarify for everyone, what we are measuring here is Facebook's reporting about, or meta's reporting rather about. Incremental attribution versus standard attribution. It's not necessarily that we've run sort of an quote unquote independent geo holdout lift study yet, but that's next on the docket.

Here, one more question I you can, we can cut this if it's not relevant, but the ROAS lift that's reported on the standard attribution, what does that a ROAS lift over?

Just the average account ROAS or something.

McCay Rasmussen: Yeah, how I understand it, it's essentially just giving the lift in, in what the end platform noise is showing.

Richard Gaffin: Gotcha. Okay. Alright, so I think that covers then sort of a, the, the challenges with our initial wave of reading incrementality. B, why we're so excited about this, and C, some initial kind of promising data [00:12:00] about what the IRS setting might be capable of. And then we've given like a little bit of a roadmap for what's next, but maybe kind of flesh that out a little bit beyond just doing the geo holdout tests.

What do you kind of see down the road? A few more steps.

Tony Chopp: Fully optimized.

Richard Gaffin: That's right.

Tony Chopp: No. I

Richard Gaffin: right.

Tony Chopp: I mean, McCay and I were sort of joking about this in these conversations. Like he McCay touched on it. Touched on it earlier about like all, all the things that media people do. We, you know, we go build campaigns, we, we make optimizations, we communicate, and, and you, you often, like in this media world, you hear like, sort of a thing.

Like, I want to, I wanna make sure our campaigns aren't fully optimized. Like, what does that actually mean practically for all intents and purposes, like, I think this. This. What this represents is that that potential,

Richard Gaffin: Mm-hmm.

Tony Chopp: so me be more specific. I have a campaign that has, I understand what I'm selling and I understand the unit economics of what I'm selling, so I can [00:13:00] get to setting a target.

Richard Gaffin: Mm-hmm.

Tony Chopp: gonna caveat me here because we don't have. We don't have any efficiency based bidding for this product yet. They're running it with just highest volume.

Richard Gaffin: Mm-hmm.

Tony Chopp: TTBD soon to come, but assume we're gonna get there. I have the target right I have the as far as audience goes, broad

Richard Gaffin: Yeah.

Tony Chopp: and, and I've, I'll take it a step further, like no more excluding anybody, right?

Richard Gaffin: Mm-hmm.

Tony Chopp: this, this incrementality tool figure out the incremental contribution of my new versus my returning customers. And, and you know, we have a bunch of different ways that we talk about returning customers. So, got my target set. to you super wide open on the audience side. Inflated budgets so I don't miss any potential opportunity, and always on incre incremental impact testing that flows back into it. And what we're talking about is truly and honestly a state of [00:14:00] fully optimized, send

Richard Gaffin: Yeah. Interesting. So it's a, a step closer than to Zuckerberg's dream of you put money in, you get money out, and that's essentially what I. What ad buying becomes. Right? Yeah. Cool. All right. Well, it's a future we can all hope for and as, as we kind of continue down that road towards figuring out whether that's coming or not I will keep in touch with both Tony and McCay about this over the next few weeks.

So, when, when do you've expect kind of the next round of testing to come in?

McCay Rasmussen: Yeah, I think like the roadmap here is, is, and, and I would encourage anyone because incremental attribution should be rolled out to most accounts by now. So I think like the first step is doing. An in platform test like we've done here you can set up the conversion lift test under the experiments tab and then just do a two cell test similar to what we did. The next path, or I guess the next steps in this path for, for me, and I'd say for anyone else is, is getting a true incremental incrementality test on incremental attribution. So doing a a, a legit geo holdout, not necessarily in platform, but through a third party [00:15:00] tool or we have our own at CTC getting a read there and then the end state is pivoting the entire account to Tony's point to incremental attribution.

So that's what we aim for.

Richard Gaffin: Awesome. All right, well, we're looking forward to it. Anything else you guys? Tony McCay want to hit on this before we wrap things up here?

Tony Chopp: Yeah, I, I've, I told McCay that we, we gotta be back here in not too long to talk about that full account running on this incremental attribution product. So we're, we're, we're pushing hard into this. I, I, I believe it's the future. I believe we're actually on the verge of something that's really, really meaningful in our industry.

And we're, we're excited to, to keep plugging away at it and testing it so we can bring it, bring it forward to our clients.

Richard Gaffin: Awesome. All right. Well, as soon as we know more, we will let all of you know as well. But until then, we will see you next time. Take care everyone.