When the AI Gets It Wrong

This is the lesson that keeps you out of trouble.

Back in What the AI Actually Knows, you learned the mechanics: the model predicts plausible next words. Plausible is not the same as true — and here's the dangerous part — the tone doesn't change when it crosses that line. A wrong answer arrives in the same confident, well-organized voice as a right one. There's no wobble to catch, no "um" before the made-up part. For a busy founder skimming output between meetings, that's the trap.

The four ways it burns founders

Invented statistics come first. "Studies show 73% of customers won't return after a bad delivery experience" — round, plausible, sourced from nowhere. Then made-up regulations: ask about Calgary's watering restrictions or Ohio's food-handling rules and you'll get a fluent, specific answer, but fluency isn't jurisdiction, and the model may be blending three cities' bylaws into one confident paragraph. Third, fake citations: real-sounding study names, real journal titles, articles that don't exist. Ask for the link and watch it apologize.

The fourth one is quieter. Call it yes-manning, or agreeable drift: ask "this pricing strategy is solid, right?" and the model tends to agree, because it's completing the pattern you started — and the pattern you started was agreement. It isn't lying. It's mirroring your framing instead of challenging it, which is exactly what you don't want from an advisor.

Sarah checks the number

Sarah is drafting a Q3 memo for an e-commerce client when the AI states, cleanly, that the client's state has a $100,000 economic-nexus threshold for sales tax. Specific, confident, plausible. Also precisely the category of claim she never forwards unverified — tax rules vary by state and change without notice, and a fractional CFO who's wrong about one loses more than an afternoon. So before touching the memo, she asks:

Two useful things happen. Asked directly about confidence, models hedge more honestly than their prose suggests — the certainty was in the writing style, not the answer. And "argue the opposite" snaps the model out of yes-manning; it went from asserting the number to explaining that thresholds like this have shifted in several states recently. Sarah spent ninety seconds on the state revenue site, confirmed the current rule, and fixed the memo before the client ever saw it.

The habits

Four habits cover nearly every failure mode:

• Never publish a number, law, or citation you didn't verify yourself. The AI drafts around facts; primary sources confirm them.
• Before relying on anything specific, ask: "What would you need to know to be sure?" The answer doubles as your checking list.
• When you like a conclusion, make the AI argue against it before you act on it. If the opposing case is weak, proceed. If it's strong, you just saved yourself.
• Apply the stakes rule: the higher the stakes, the more human review. An Instagram caption gets a skim. A price change gets a careful read. Anything touching taxes, law, safety, or money gets verified line by line — by you or a professional.

None of this means the tool isn't worth trusting. It means trusting it the way you'd trust a bright new hire: excellent first drafts, genuine blind spots, and never allowed to sign anything on their first week. The founders who get burned aren't the skeptics or the enthusiasts — they're the ones who stopped reading.

Try it now

Grab something the AI told you this week that included a number, a rule, or a citation. Run the verification follow-up from this lesson, then spend five minutes finding the primary source. One of two things happens: you confirm it and trust your process more, or you catch it — and quietly become the person in your industry who doesn't publish other people's hallucinations.