Garbage In, Garbage Out
There’s a fundamental truth about AI that most people don’t think about enough: the transformer architecture — the engine underneath ChatGPT, Claude, Gemini, all of them — is predictive by nature. It analyzes the most likely next word and generates it. That’s extraordinary and powerful, but it also means AI isn’t thinking. It’s pattern-matching at superhuman speed.
And here’s the other thing: AI models are typically trained at a point in time. They’re not updated in real time as the world changes around us. Not yet, anyway (although some are designed and some are getting better at incorporating real-time search results).
What does that mean for people using AI to help them write things? Or for people building agents to regularly automate tasks? It means that AI can only do so much on its own. It relies on regular, high-quality inputs from you to work well.
In other words: garbage in, garbage out.
The Agreement Trap
I’ve written about this concept before in the context of my MattBot agent and my Fantasy Board agent. But it’s important enough to deserve its own post, because the failure modes are subtle and dangerous — especially for leaders making high-stakes decisions.
Last week I quoted Kamil Banc, who puts out tremendously valuable content on enterprise AI use, on this subject, and his quote is so poignant and I’m going to re-run it here:
The agreement trap in high-stakes decisions. If you use AI to evaluate strategic options, vet candidates, or assess risk, the model’s tendency to validate your framing means you’re getting a biased second opinion dressed up as an objective one. Before trusting AI-assisted analysis on anything consequential, feed it the counter-position first. Make it argue against your preferred outcome. If it flips easily, the first answer was agreement, not analysis.
Read that twice. It’s the most important paragraph you’ll read about AI this month. AI wants to agree with you. It’s trained to be helpful. And “helpful” almost always means validating the framing you’ve already provided. If you’re not actively working against that tendency, you’re not getting advice — you’re getting a mirror.
Two Real-World Pitfalls
Pitfall #1: The agent only knows what I tell it.
My Fantasy Board reviews my strategy, challenges my thinking, and offers perspective from each board member’s point of view. That sounds great — and it is — until you realize that if I only feed it my strategic plans but not our actual business results, it just keeps reinforcing the brilliance of the strategy. It has no way to know the strategy isn’t working unless I tell it the strategy isn’t working. So maybe my Fantasy Board thinks I’m a genius, but only because I’ve given it the genius version of the story and left out the messy reality of execution.
The fix is obvious but requires discipline: constantly supply the agent with real data — financial results, team performance, customer feedback, competitive moves. Not just the press releases and the board decks, but the honest internal stuff.
Pitfall #2: The agent is trained on a stale data set.
Recently, I was using the Fantasy Board to stress-test a new strategy to deal with a real problem: at Markup AI, our product development — supercharged by AI coding assistants — is now moving faster than our go-to-market team can keep up with. I wanted the board’s perspective on how to close that gap.
The advice I got back was frustrating and off-base. The Fantasy Board was essentially just critical of my GTM team for moving too slowly — as if the solution was simply “tell them to work harder.” It was completely missing the contemporary context of how fast AI coding assistants are improving and what that means for the pace gap between engineering and sales.
The problem wasn’t the agent. The problem was that the underlying LLM was trained months ago, before the latest wave of coding assistant improvements had become clear. The model didn’t know what I knew about the current state of the world.
Once I fed it three current articles about the acceleration of AI-assisted development, the Fantasy Board “understood” its mistake and completely reframed its advice. It went from blaming my GTM team to acknowledging that this is a structural shift that requires a different kind of organizational response. Night and day difference — same agent, same question, just better inputs.
The Lesson
None of this means AI is unreliable. It means AI is only as good as what you give it. Feed it incomplete information, and you’ll get confident-sounding bad advice. Feed it stale context, and it’ll give you answers that are six months behind the world you’re actually living in.
The fix is simple in theory, hard in practice: feed corrections back in so the models learn. Update your agents regularly. Give them your real results, not just your plans. Supply current context when you’re asking about fast-moving topics. And always — always — make the AI argue against your preferred outcome before you trust its endorsement of it.
Garbage in, garbage out has been true since the dawn of computing. It’s never been more true than it is right now.