Why Meta's AI Has No Point of View

[00:00:00] Richard: Hey folks, welcome back. So this week, we're sharing something from Taylor that I think is one of the clearest explanations of why the context layer matters so much when you're building with AI. He's gonna walk through a specific example where the same tool goes from completely wrong to genuinely useful just by giving it a framework for how to interpret the data.

[00:00:16] So if you've ever been frustrated by AI hallucination, or maybe more importantly, if you blindly trust your AI output, this is a critical listen. Here it is

[00:00:26] Taylor: Uh, I wanna g- I wanna give you an illustration of why this idea of the context layer, um, for your AI tooling is so critical and is a differentiator in all of this. And this goes to ... I was on a call, I'm part of this catalyst group that Meta uses to pre-release features to try and generate PR buzz amongst the sub- subset of influencers.

[00:00:44] And, um, on this call, they were trying to, they were debuting Monos to us and, um, someone asked a question and they asked like, "What is the methodology that, uh, Monos is trained on to make media buying recommendations?" And the answer was really telling. They basically said back to them, "There is no sort of underlying training of methodology.

[00:01:02] It's there to do what you ask it to do. " And it was very clear that they're uninterested in sort of being the, uh, answer engine and more about the execution function of your ideas. And this is sort of what you see a lot in AI, is that it will hallucinate or make recommendations relative to the quality of the context that you give it for how you want it to interact with you.

[00:01:23] And I'm gonna illustrate this for you. So this is a, uh, screenshot out of StatList that gives context expectation for a brand of ours performance month to date. And I took that screenshot and I gave it to my Open Claw instance, um, and, uh, I said, "Build me an interactive HTML analysis of the right action to ensure we hit our contribution margin goals for this month.

[00:01:43] How should I prioritize the actions?" Uh, this is a thing at CDC, like we only build an HTML now. Everything's sort of decks are gone and so this is a common request. Um, and you'll say build, you see it responds that says, "Building here, here's the quick read while the dashboard assembles." So it recognizes that I'm behind on the contribution margin goal.

[00:02:02] Um, but what you'll see very quickly is it gets to like the problem is almost entirely Google. Spend is up 51%, revenue down 45%. So if you look, that's like not actually true at all. Like Google is actually well ahead of its target. It's all green. So it just sort of hallucinates this issue somehow. Um, in part because it doesn't really understand the context of the UI of the dashboard and, um, the numbers and how to interpret them.

[00:02:27] And then it says Meta's carrying the business and it's sort of like, oh, hmm, uh, then it starts to build the dashboard. And so I intervene and I'm like, "No, no, no. Google is ahead on spend and efficiency. It's not a problem at all. You can see here Google's well ahead of its IRO's target, beating revenue and spend, doing quite well."

[00:02:42] And it says, "You're right, I misread the color coding. Green equals good, red equals bad. Let me reread." Google's actually outperforming. So this is like this experience people have that's very frustrating with AI, right? Like it sort of hallucinates relative to what you want it to do. You want it to intuit the question that you're asking and be able to respond.

[00:02:56] Um, but, uh, then I say, okay, and this is where the context really matters, is I want you to watch this video and apply the knowledge of your analysis to the screenshot. Um, and this is a video of mine that's called the hierarchy of metrics where I go through, basically the way that this dashboard was designed is to help our people and customers sort of sequence through a large set of information in priority.

[00:03:20] So contribution margin is the largest number. It's the goal. That's why it's there on the top. And then there's this sort of cascade of importance of the metrics as you move down the set of information. And so this video explains that. It's the methodology that overlays the data structure, um, and the display here in this case.

[00:03:39] And what you'll notice is being able to have that context, being able to provide that methodology layer, here's what I get in response after. "Good, I watched it. Your pyramid of success framework changes the entire analysis. Let me redo this properly top down. Scoreboard, contribution margin. This is the only number that actually matters.

[00:03:55] Everything below exists to diagnose and protect this number. "So right away, now all of a sudden, it's behaving and enacting a methodology perfectly, but only once I've given it the context for how I want it to behave. Um, it goes through business orders, AOV, the scoreboard gap is small, customer metrics layer goes through and talks about new customer acquisition.

[00:04:14] The sponge is not being drained. New customer acquisition is running hot, but that's good. This reframes the entire CM gap, channel metrics, real diagnosis, the CM gap is in a crisis. It's a small miss while growing new customer acquisition. Channel metrics performing, customer metrics healthy, business metrics below plan, scoreboards there, the right action in pyramid order, and it goes through and then it ultimately, um, built this entire like sort of AT- HTML action plan, uh, sort of with interactive sliders, et cetera, et cetera.

[00:04:43] Um, and, and all of that is just an illustration to say that without this context, without the ability to provide it a way in which to interpret the information, the information was basically useless. And I see a lot of people having this experience with AI right now where the result is like the numbers are wrong.

[00:05:02] It's hallucinating. It's lacking the ability to be impactful. Um, and what I've found is that we're experiencing something very different, which is that because we have such a rich context layer, there's methodology for everything, and that's where we're spending a lot of energy. We're able to give it really well structured data in light of that methodology, and it becomes a perfect execution of the ideology.

[00:05:23] Um, and so this is a common conversation. There's a great article that, uh, w- w- By Roland's the other day talking about this idea that the context layer is everything in businesses. It's what is your revenue definition? And is it gross sales or is it net sales? How do you interpret in light of that? How do, what do you, how do you define contribution margin?

[00:05:41] What do you want to be included in that means? What are your brand standards? What are your things that you would never want to do? What are the elements of your business that provide, uh, the raw material and lan- allow it to translate in the way that you want? And so, I think in many ways, this is the structural advantage that we create that allow our AI inside of CTC to compound and value quickly is because we have this, uh, sort of long, rich history in the form of everything from YouTube videos to blogs to internal docs, et cetera, to a, a data visualization system and forecasting process that allows it to continue.

[00:06:12] So, um, curious if you all are experiencing any of this context, uh, application. It also is what makes me concerned for, uh, tools like Monos or others is, is that they, uh, I've said this many times, they're not designed to have a point of view. They're designed to help you, um, uh, en- enact your desires. And I think it's gonna be very unlikely that Meta provides a point of view, um, because it will lack the context.

[00:06:36] It doesn't have your underlying business objectives in mind. And when I think about RL, like this idea of reinforcement learning, um, these tools are incredibly good when there's an objective R to RL against, right? And so just even defining something like contribution margin as the primary driver, um, against which you want to RL is like really important.

[00:06:56] Now, if you watch the hierarchy of metrics video, we know that that played out, um, in isolation would drive you to, uh, squeezing the sponge, driving a bunch of existing customer revenue. And so if you are out against a single objective like that, then you'll miss new customer acquisition. And that's why a framework like the hierarchy of metrics and the ability to think about the long-term health fund organization relative to enterprise value and these other things become the way in which you can get to these objective measures for the systems to RL against, uh, within the context of that framework and the business objective.

[00:07:26] This is why I, I talk a lot about that the primary problem for organizational design, whether it's the, uh, org chart or now underlying AI impact is about the lack of clarity of the founders to drive the clear and obvious endpoint for the company that everything should flow out of. Uh, and that only gets extrapolated in a world of AI where it will run in almost any direction.

[00:07:46] And so if you're experiencing this sense of it is hallucinating or it's wrong constantly, I wonder if the objective is clear enough to RL against and if some sort of institutionalized frameworks, um, might be a thing that you would benefit from, uh, or that a partner like us could help you bring to life.