How We Forecast $4B in GMV to 3% Accuracy

[00:00:00] Luke: So the philosophy that underpins forecasting for us is that all models are wrong, the best ones are useful.

[00:00:06] And the point of a forecast is not to predict the future perfectly. You will be wrong. Every forecast is wrong. The point is to understand where you are wrong so that you can quickly course-correct before any damage is done or to capitalize on the opportunities that you're seeing. If on the second or third day of the month you can figure out which input on your model is too high, too low, or unexpected, you have time to fix it.

[00:00:26] All right, everyone. Welcome to the "Ecommerce Playbook" podcast. I'm going to be walking us through a new series that we are kicking off on the CTC Canon. The CTC Canon is something that has been some months in the making, but really over 12 years of our experience in the e-commerce world has informed what the Canon is for us, which is the operating principles and how we approach growing direct-to-consumer e-commerce brands profitably.

[00:01:01] We've built the Canon across a number of dimensions of the key disciplines that any that any brand needs to be able to operate against, from forecasting to meta media buying to creative strategy, email strategy, media measurement, and we're gonna be walking through each of these things in sequence and doing a, a bit of a kimono reveal of how we approach doing what we do that's based on a few billion dollars in GMV and over 12 years of, of experience doing this across a lot of brands.

[00:01:34] Which is the unique perspective that we have in this world is that we have a larger data set, we have a lot of reps across a large number of brands, across a number of sizes, verticals, industries, that helps us to see what is effective, what's not as effective. And the, the thing that we have been focused on is as much as we can, building that into a co- a cohesive operating principle or canon that it-- that serves as the basis for how we work with brands that come to us.

[00:02:04] This allows us to be effective even when new people are in are in the works new people working with the brands at CTC. Whether someone has two years of experience or 12 years of experience in the space there, there is the, there is the this cumulative knowledge informed by the canon that allows us to compound this kn- knowledge over time that amplifies our effectiveness for the brands that we work with.

[00:02:28] So it's an ever-evolving canon. It is something that right now we have built this current version, and it will continue to evolve based on what we see across our data set. But we're gonna walk through the canon as it exists in sequence, and today we'll be focused on forecasting. Forecasting is what forms the foundation for the execution across all the other disciplines that we are going to work through.

[00:02:54] And what we believe is that great forecasting is an exercise in execution as much as it is in modeling. So, the forecasting section that I'm gonna walk through for these next few minutes here is how we combine propri- proprietary data science with daily operation discipline to make profit predictable.

[00:03:15] Over our-- across our data set we achieve three, a little over 3% in forecast accuracy across the brands that we work with. What we'll say is that the error bars that are acceptable as it relates to forecast accuracy are, are plus or minus 10% in either direction, right? If we massively under or over forecast, it is an unhelpful forecast for us to be, to be able to make the business decisions that we need to related to the media allocation, media measurement, the inventory purchasing, et cetera.

[00:03:44] So 3% forecast accuracy across our data set. We forecast it across $4 billion in GMV and that is how we operate, is getting predict-- profit predictable in what our forecast is, is able to achieve. So the philosophy that underpins forecasting for us is that all models are wrong, the best ones are useful.

[00:04:02] And the point of a forecast is not to predict the future perfectly. You will be wrong. Every forecast is wrong. The point is to understand where you are wrong so that you can quickly course-correct before any damage is done or to capitalize on the opportunities that you're seeing. If on the second or third day of the month you can figure out which input on your model is too high, too low, or unexpected, you have time to fix it.

[00:04:23] If you get to the end of the journey and realize you're off course, there's, there's no time to fix it. So that is what makes a forecast useful. And a a quote from George E.P. Box, who's a statistician. He says, "Since all models are wrong, the scientist must be alert to what is importantly wrong.

[00:04:40] It is inappropriate to be concerned about mice when there are tigers abroad." So we're the, the point of the forecast is to understand where we are off course, so we can course-correct and to pri-prioritize that against the highest impact actions that we can take to move the business. What does success in forecasting mean?

[00:04:57] It means that there's a close approximation of reality. Again, plus or minus 10% to target is an effective forecast. There's early detection of deviation. So a great model has an expectation of every input every day, allowing you to course-correct along the way. That's what makes a model useful. And then the focus on ti-tigers, not mice.

[00:05:15] So bad gross margin is a tiger. Bloated OPEX is a tiger. The next headline variation on your ad creative is a mouse. How can we prioritize the highest impact levers that are going to drive the business forward? So that is the philosophy. The system that underpins forecasting for us is what we call an operating model for profit, profit.

[00:05:36] It's not a spreadseet-sp-spreadsheet that spits out a number. It's an entire system for managing and growing your e-commerce business. It involves four steps that transform qualitative planning into daily operational precis-precision. The first step is qualitative planning. This is designing a 12-month marketing calendar, every promotion, launch, and cultural moment.

[00:05:56] If you're familiar with anything that we talked about-- talk about, marketing calendars is at the center of the forecast and the planning exercise. Then step two is quantitative modeling. Three proprietary data science models predicting spend, revenue, and retention are built off the base of the historical data.

[00:06:13] The third step is building a map. Every dollar, every day, every campaign laddering up to the financial goal. And then the fourth step in this operating mof-model is plot, pivot, and profit. It's a daily cadence track-tracking actuals versus the expectation, identifying, acting, and course-correcting along the way.

[00:06:32] Qualitative planning, quantitative modeling, build the map, plot, pivot, profit is the model that we operate against the forecasting philosophy This brings us to what we'll dive into as it relates to specifics around each of those four steps. So first, on the qualitative planning. Forecasting begins with planning, not math.

[00:06:57] What we see is reality for e-commerce brands at large and even outside of that, is that revenue is n- never linear. In almost every business, moments drive outsized performance across the calendar. Black Friday is an obvious moment, but every brand has at least one other natural peak built into their cultural calendar, and the best brands ma-manufacture additional ones throughout the year.

[00:07:19] And that's even before we get to product launches, category launches, et cetera. These peaks work because of how ad platforms price efficiency. When you drive unexpected increases in conversion rate through promotions, product releases, or cultural moments, you arbitrage Face-Facebook's auction formula.

[00:07:35] Your best ROAS day is never an evergreen ad. It's a moment of urgent purchase demand where conversion rate spikes and you get outsized value return. So the process starts here, which is we build a 12-month marketing calendar with every email, promotion, product release, and cultural moment plotted, identify the peaks, fill the valleys, and design the revenue shape before a single number enters a model.

[00:07:56] We ingest two years plus of marketing calendar data against every single one of these moments, the email, SMS sends that surround each. That serves as the data foundation against the, against these, that informs every marketing moment. Everything on the marketing calendar is tagged by s- a specific type, which allows us to create structure around the marketing calendar, and then we build that out into the future as well.

[00:08:19] And so the, the in- data infrastructure surrounding the 12-month marketing calendar then helps to inform the future expectation of the moments that we are planning out against those areas. Once we have the foundation laid of the qualitative, we move on to the s- second step in the operating model, which is quantitative modeling.

[00:08:39] So there are a, a few core models that inform the quantitative modeling in this process. We build customer-centric, customer cohort-specific forecasts, not channel trends, not year-over-year extrapolation. Every forecast is built through the customer file because cohorts are the atomic unit of e-commerce.

[00:08:59] There are many different ways in which forecasting and modeling is approached by e-commerce brands, and we've seen them all over the 12-plus years, over the hundreds of brands that we've been able to work with, from channel-level forecasting to visitor conversion rate, AOV expectations. And we center around customer cohorts because that is what's driving the revenue.

[00:09:19] At the end of the day, you have the, you have your core customer cohorts of new and returning customers that are driving the revenue. They are the atomic unit of e-commerce that sit at the center. So the first model that sits in this quantitative modeling section is the spending power or, or what we've referred to historically sometimes as the AMER model, so the spending power model The spending power model analyzes the relationship between media spend and new customer acquisition efficiency, or AMER, over time.

[00:09:46] We observe all of the brand's historical data, then we clean out the data for ano-anomalies, unrepeatable events, product launches, one-time spikes that won't occur, so that we have a clean data set that can be predictive and informative about the future. This produces a reliable spend to efficiency curve that predicts new customer revenue at any spend level, giving us the first building block of expected monthly revenue.

[00:10:09] Spending power AMER model sits at the intersection and the first step of the forecasting process, allowing us to understand what is the optimal level of spend that we should operate at based on the stated business objective. So we orient around three core business objectives in terms of optimization selection.

[00:10:27] The first being that we wanna maximize contribution margin from new customers in month one. So what is the level of spend based on my historical degradation efficiency that's informing this model? What is the level of spend that it should operate at that's going to maximize contribution margin from my first-time customer cohort in the stated time period that accounts for seasonality and where we're at in the year?

[00:10:50] The second optimization is what is the optimal level of spend for maximizing new customer revenue at break-even contribution margin? So I wanna maximize the amount of, of revenue that I'm driving from new customers at a break-even contribution margin. What's that level? And then the third optim-optimization being maximize contribution margin against the stated LTV or our, our lifetime defined period.

[00:11:13] So I wanna maximize contribution margin for my first-time customers, but I'm willing to wait 90 days to realize that impact. And so each of those optimizations at a different point in the degradation curve for each brand based on historical data, and then we are able to see the specific trade-off, quantify the specific trade-off of the additional $10,000 in spend or the additional $100,000 in spend.

[00:11:36] What is the anticipated return of revenue, and what is the trade-off in contribution margin as a result against that to be able to inform the optimal budget allocation and the new customer revenue surrounding that first customer cohort of new customers? The second model in this process is cohort LTV The core LTV mo-models the retention curve of your customers over time to predict returning customer revenue.

[00:12:01] We track cohorts monthly, measuring active versus lapsed customers. Lapsed is defined at the point where 80% of second purchases have occurred. A growing active customer file predicts revenue growth. A sh- a shrinking one signals trouble. No matter how large the total file looks, many of those customers are lapsed and dead.

[00:12:19] They are not coming back. The retention model-- What's important to understand about th-the returning customer cohort is it-- there's really two subsets of your, of your returning customers, even before getting to the the lapsing point. The first is you have all the existing customers that you acquired up until now.

[00:12:36] So all the customers that have come and bought from you at some point in the past that live in your CRM, you have email contact for, et cetera that might return in the future. You have a second cohort, which are all the new customers you're going to acquire two months from now, four months from now, six months from now, who have an expected repeat behavior on off the back of those customers as well.

[00:12:59] And so the cohort LTV model is focused on us being able to project the predictive contribution of all the existing customers at this point in time, as well as the new customers we're gonna acquire in the future based on the spend AMR model or the spending power model and their anticipated impact, so that we can get a full view of all these current customers, the new customers we'll acquire in the future, their contribution, what is the expected new customer and returning customer revenue and orders over those periods of ti- over those periods of time that we can ser- that serves as, as the foundation of the forecasting process.

[00:13:36] Once these two first models are built, then we get to the third, which is the event effect model. The event effect model is focused on helping us to quantify how marketing moments, emails, product releases, promotions affect both the new and returning customer revenue on any given day. So this is, this is the connective tissue between your qualitative planning and the quantitative output.

[00:14:00] Your marketing calendar becomes a mathematical input, not just a content schedule. Each event carries a measured performance multiplier derived from historical data. So as I mentioned in the marketing calendar process, we are ingesting historical marketing calendar events, tagging each appropriately to its marketing event and action type.

[00:14:20] And that serves as now a, a layer of data that can inform these qualitative actions in a quantitative way, which is critically important when you have very different relationships of these marketing events on impacting the revenue from your different customer cohorts. Product launches, for example, for brand A, product launches may drive a substantial increase in your returning customer revenue on the immediate days surrounding those launches, right?

[00:14:47] Because we're launching on the site, primarily pushing through email, SMS and our, and our own channels, and our existing customers are interacting with those product launches in a really strong way. Where for-- on the new customer cohort side of things, that new product actually may take some time in market to be able to drive the new customer behavior.

[00:15:08] Inversely, a promotion may have the impact where it's gonna drive the returning customer revenue but in addition, it's going to increase the conversion rate on any new, new traffic that's being driven to the site and the new customer cohorts that are, that are coming through. And we can go on down the line.

[00:15:23] Every brand has a different relationship of marketing event types and the impact on their new and returning customer cohorts on the specific days when those marketing moments are planned and the actions drive that behavior. So the event effect model is the important connective tissue between the spending power model, the core LTV model, and then the marketing calendar that allows us to, off the foundation of those two customer cohorts models, be able to project the impact of these marketing calendar events over time.

[00:15:53] These three models together produce a monthly expectation of spend and revenue. We then layer in all historical costs, variable and fixed, to build a p- full P&L level forecast for the business. So we integrate on the variable cost side of things COGS, delivery costs, et cetera. All of that is ingested.

[00:16:11] We layer on fixed costs and OpEx to get to a p- full P&L level forecast that then builds out from the foundation of these models. From there, we use an MMM to allocate the monthly media expectation into channel level and campaign level daily targets that ladder up to the overall financial goal of the organization.

[00:16:29] Again, we're not starting at this campaign operating and driving this much traffic at this high of a conversion rate with this AOV value to get us to a revenue forecast. We are starting with a financial goal that's driven by the customer cohorts, the atomic units of e-commerce, and then backing into what the channel level expectation needs to be to be able to achieve that financial goal The core thing that we are forecasting in this exercise is contribution margin, not vanity metrics.

[00:17:01] We provide a 24-month P&L forecast all the way down to net profit, but the model is built around driving contribution margin because it represents the closest proxy to profit where we control all of the variables. There are only two-- ever two kinds of problems to solve volume or efficiency problem. So every deviation from the forecast falls into one of those two categories, and each has a specific set of corresponding actions.

[00:17:24] When we are looking at the business pacing against forecast at any given point in the month, day two of the month, day 17 of the month, we are starting by looking at how is our contribution margin pacing to expectation. That is the that is the tiger in this situation. We are not starting with what does our our attribution-adjusted meta ROAS look like against target.

[00:17:46] That is, that is that is what we will get to in the sequence once we get to once we have alignment on the contribution margin pacing to forecast against that expectation. And when we're looking at our contribution margin forecast, there are only ever two problems to solve: volume or efficiency.

[00:18:04] Are we not driving enough volume, or are we not being effi- as efficient as we need to be to b- be able to achieve this outcome? Oriented around contribution margin then allows us to isolate whether on revenue we're not we're not able to-- we're not driving as much volume as we need to, or spend, and get to the channel-specific level to be able to understand what changes need to be made, or related to the efficiency MER, AM AMER, the ROAS metrics, if that if those are the challenges to sort out.

[00:18:32] But contribution margin stands as the most important thing that we're tracking against and serves the, the decision-making on a day-to-day basis. Which gets us to the final portion of this operating model against forecasting, which is plot, pivot, profit, daily execution, every dollar, every day, every channel.

[00:18:50] So these monthly goals that we build in this forecasting process are broken down into daily expectations for every channel. Every ad dollar, every email send, every campaign has a target. We also model the exact amount of creative production required, so your team can align the creative supply chain to the plan.

[00:19:06] We'll cover that in the creative section of this Canon series. But on the first day of the month, you see exactly where you are relative to e-e-expectation every single day. So we have over 35 metrics forecasted out for every single day of the month that ladder up to the yearly target And that allows us to quickly identify where we are off course.

[00:19:27] Each morning, we plot actual performance against the daily expectations in, in in our system, in Statlas, and we're able to see where did we land against the expectation for that specific channel, for that specific metric. We identify where the deviation is. Is it volume or efficiency? Diagnose the root cause, losing cash, shift budget, adjust creative and then profit.

[00:19:48] Ultimately, we take the corresponding actions while we are adjusting and identifying them in real time to be able to impact the forecast for the month. Every campaign across every channel has a daily spend target, projected ROAS, actual performance. Moments are tagged so we can see the effect of each marketing moment in real time, and we can flag for deviations immediately based off of this system that we've built.

[00:20:14] And what our team does, our profit engineers, is we follow what, so what, now what format, where each day we're sharing where we landed relative to the forecast. So the what, what is the data showing us? Yesterday's actuals versus daily. The so what, what does that mean? Is this a volume or efficiency problem?

[00:20:30] Which input is off? And then the now what. What are we doing to execute against the forecast? And the what, so what, now what built against the system and the operator, the profit engineer identifying and taking action against those is what allow us to be less right about the forecast and more useful in hitting it.

[00:20:47] We have a very robust forecasting and modeling and data ingestion process, but the focus for us still is on the execution against that plan. And it's only by being able to have the system built out and transparency into each one of the expectations against every single one of these 35 marketing business metrics to be able to understand where off course and adjust in real time to ensure that we land within 3% to forecast accuracy against revenue and contribution margin and drive across our portfolio of brands 32% average revenue growth and 41% average contribution margin growth.

[00:21:23] It's an exercise in execution more than it is in planning or forecasting. Remember, all models are wrong. The best ones are useful. This is how we approach forecasting. This is the first section on our Canon series, and we will dive into the next one as we sequence into each of the other discipline areas and how we approach these to drive growth for e-commerce brands.