The Cost of Working with Those Who Believe Themselves Too Much

force them to surrender authorship, status, and the right to say the answer came from them. That is why so much corporate life is spent defending familiar language rather than testing mechanism. It is also why the age of AI will not mainly reward the teams with the best prompts, the best demos, or the best self regard. It will reward the teams that learn to separate truth from origin before their competitors do. The prevailing belief sounds responsible enough to survive most meetings. Judgment, in this view, is safest when it stays close to experienced people, approved channels, and data gathered inside accepted systems. Outside input can be useful, but only after it has been translated, framed, and made legible by the people who already own the decision. That belief does not present itself as vanity. It presents itself as prudence. It says that caution protects quality, that provenance protects accountability, and that unfamiliar output deserves extra suspicion because the cost of error is real. That case deserves to be taken seriously because part of it is true. High consequence decisions do need scrutiny. Model output can be wrong. Analysts can overstate what the data proves. Managers can talk themselves into certainty that was never earned. Regulated businesses cannot act on a hunch because the hunch is elegantly written. The case for discipline is not weak. The problem is where most institutions place the discipline. They place it on origin before mechanism. They ask who produced the output before they ask what would have to be true for it to be right. They trust the familiar too early and distrust the foreign too eagerly. Then they call the result judgment. “What would have to be true for this outcome to keep repeating.” That is the question Feynman forced into the Challenger investigation, and it is the question most firms still avoid because it strips belief down to mechanism. It does not care whether the claim came from a decorated executive, a seasoned engineer, a vend or deck, an operating review, or a machine. It asks whether the causal story survives contact with reality. It asks what physical, operational, or economic conditions must hold for the result to occur again. It asks what would break the claim. It asks what evidence matters and what merely flatters the current narrative. That is reasoning first. Not reasoning instead of evidence. Reasoning that puts evidence back under the rule of consequence.

The wrong question shows up first

Most teams do not begin with that question because most teams are organized to defend continuity. They are rewarded for sounding careful, for respecting process, and for limiting exposure. In that setting, the first question an unfamiliar answer triggers is rarely what might be true here. The first question is usually some variation of who produced this, why does it not sound like us, and what risk do I absorb if I let it stand. The language of rigor often arrives later, after identity has already taken the wheel. This is not a moral failure. It is a well documented feature of human judgment. Kahneman’s work on intuitive and deliberate thinking did not argue that people are foolish by default. It argued that judgments often begin with fast impressions and only later encounter slower checking. System 2 does not arrive first in most real situations. It arrives after an answer has

already made itself attractive, coherent, or socially acceptable. The discipline problem inside organizations is that the second pass is often weak, rushed, or selective. It is used to justify the first answer, not test it. That helps explain why confirmation bias is so stubborn in professional life. Once a team has framed the issue in a way that protects its own prior view, incoming evidence is processed through that frame rather than against it. Reviews of professional decision making show the pattern across management, finance, medicine, and law, with overconfidence recurring more than most leaders would care to admit. Expertise does not remove the danger. In many cases it gives the danger better manners and more persuasive vocabulary. This is also why authorship matters more to organizations than they admit. Not Invented Here is not just a complaint about politics. Research describes it as a persistent decision error rooted in bias against external knowledge. The external idea is discounted not because it fails the test, but because it arrives from the wrong side of the boundary. That boundary might be another department, another company, another discipline, or a model. The label changes. The reflex does not. The danger is not error alone. The danger is error that feels familiar enough to inherit. AI is about to make this reflex far more expensive. At first, the output feels foreign, so teams attack it harder than they attack human work. That pattern has already been studied. People become more likely to reject algorithmic recommendations after they see the system make a mistake, even when they also see it outperform a human forecaster. The machine is not merely judged. It is judged asymmetrically. Human error is tolerated as part of the cost of doing business. Machine error becomes proof that the source itself should be distrusted. Then the second phase arrives, and it is worse. The output becomes familiar. The system gets good enough, quick enough, and fluent enough that people stop noticing where their own thinking ends and the system’s phrasing begins. At that point, the risk is no longer simple distrust. It is lazy adoption. The team that would not accept a model’s draft six months earlier now absorbs its language without checking the mechanism underneath it. The danger moves from rejection to merger. That is where judgment can rot while everyone in the room still feels intelligent. If this sounds abstract, ask a plain question at the next review. When a recommendation shows up on a page, do we test the mechanism with the same severity regardless of whether it came from the CFO’s analyst, a plant engineer, a consultant, or a model? Or do we apply one standard to the familiar source and another to the foreign one? If the answer depends on origin, then the institution is not yet serious about truth. It is serious about belonging.

Challenger was not a data problem

The enduring value of Feynman’s role in the Challenger investigation is often retold in a softened form. He is remembered as the brilliant dissenter who cut through bureaucracy with common sense and an ice water demonstration. That memory is directionally right, but it misses the sharper point. Feynman did not defeat the institution with no evidence. He defeated institutional self protection by forcing evidence back into contact with physical behavior. He would not let managerial confidence outrank the properties of the seal. The Rogers Commission did not describe the launch decision as a close call made under perfect information. It said the decision to launch was flawed. It said the people who made it were unaware of the recent history of O ring problems, unaware of the contractor’s written recommendation against launching below 53 degrees Fahrenheit, and unaware that engineers continued to oppose the launch after management reversed position. That is not a story about missing data in the abstract. It is a story about what happens when filtered information reaches authority in a form that protects a decision path rather than tests it. Feynman’s Appendix F remains devastating because it names the gap that institutions prefer to blur. He described enormous differences between engineers’ estimates of shuttle failure risk and management’s. He asked what caused management’s fantastic faith in the machinery. The phrase matters because it captures a pattern that goes well beyond aerospace. Organizations often confuse repeated operation with validated safety. Success becomes anesthetic. Each prior non failure is counted as evidence that the system is safer than it is, even when the underlying mechanism remains exposed to the same conditions that could still bring it down. The physical mechanism at issue was not mysterious. The commission found that a warm O ring, once compressed, returned to its shape far faster than a cold one. It reported that an O ring at 75 degrees Fahrenheit was five times more responsive than one at 30 degrees. Cold slowed the seal’s ability to respond to joint movement. That mattered because the seal did not have all day to become effective. It had to work in time. The joint could not wait for management language to catch up. That is the part most companies still miss. Time is not a background condition. Time is part of mechanism. A seal that works too late fails. A customer recovery that arrives too late fails. A supply chain intervention that gets approved too late fails. A machine learning recommendation that requires five layers of review before action fails, no matter how elegant the analysis looked on the slide. Firms keep treating delay as a governance cost when, in many cases, delay is part of the failure mechanism itself. Ask the next question plainly. When your team reviews a recurring operating miss, are you studying the event or the conditions that made the event likely to repeat? Are you measuring whether the recommendation was well written, or whether the action arrived before value leaked out? Are you auditing explanation, or are you auditing closure? If the answer stops at explanation, then the firm is still paying people to narrate the wound after the blood loss has already become expensive.

A result that arrives too late is not a result. It is a record of delay.

What organizations really trust Executives often say they trust data. What many organizations actually trust is sanctioned interpretation. The spreadsheet is acceptable when it comes wrapped in the right function. The model is acceptable when it reproduces the hierarchy’s preferred tone. The analysis is acceptable when it lets the institution keep feeling like the author of its own correction. This is why so many companies can be surrounded by measurement and still remain slow at the only point that matters, which is the time between signal and action. That slowness gets disguised by respectable words. It is called diligence, governance, alignment, review, escalation, and risk control. Each term sounds defensible in isolation. The problem begins when those terms harden into architecture and the architecture keeps charging the business for time as if time were free. A CFO can tolerate many forms of waste longer than he should. Time waste is the one that hides best because it enters the books through other labels. It shows up as expediting cost, churn, discounting, overtime, scrap, rework, service credits, excess inventory, write downs, working capital drag, and the permanent labor needed to rebuild context each time a decision crosses a boundary. This is where AI creates a dangerous illusion. Because it can produce analysis quickly, leaders start to think the firm itself is becoming faster. Often it is not. The model generates a draft in seconds, but the institution still requires people to re explain the problem, re win permission, and re enter the same conclusion into systems designed for a slower age. In that setting, AI does not remove the burden. It narrates the burden with more polish. The enterprise looks smarter while staying late. The deeper issue is belief. Teams believe most strongly in the things that confirm both their priors and their status inside the room. That is why the strongest discipline in the next few years will not be prompt skill. It will be the ability to convert any meaningful output into a testable claim before the institution decides whether it likes the source. The question is not whether the output flatters the hierarchy. The question is whether the mechanism survives inspection. There is a fair counterargument here, and it matters. Some leaders will say this line of thinking invites recklessness, especially in regulated or high consequence settings. They will say that unfamiliar machine output must be treated with suspicion because accountability still lands on humans. They are right to reject blind trust. The harder point is that blind distrust is not discipline either. It is merely ego with a safety badge pinned to it. The standard cannot be trust the model. It cannot be trust the incumbent expert. It has to be pressure test the claim. That standard changes the human role in a way many firms are not yet ready to admit. The highest value human contribution is moving away from first pass authorship and toward disciplined correction. That does not make people less important. It makes them more important where it counts. Not as gates that every answer must pass through because the hierarchy

demands ownership, but as the compounding mechanism that improves the quality of the system, the speed of learning, and the fitness of action over time.

The teams that learn faster will inherit the edge

The next wave of advantage will not come from who has access to a model. That will not hold for long. It will come from who builds a better discipline around revision. Teams that insist every useful answer must originate with them will learn slower than teams willing to extract signal from imperfect external output, improve it, and redeploy it at speed. That is as true for human ideas from outside the function as it is for machine generated reasoning. This is why the old pride in authorship is becoming a liability. It was easier to protect when reasoning was expensive, scarce, and tied to credentialed people. As reasoning becomes abundant, the institution that still treats cognition as private property will create a tax on itself. It will spend senior time relitigating drafts that should have been improved once and folded into the system. It will call that prudence. In earnings terms, it is overhead. Here is the prediction that will matter. Over the next three years, the firms that make AI pass through heavy human authorship rituals before it can affect action will not reduce knowledge work nearly as much as they expect. They will add review layers, not remove them. Their most expensive people will spend more time validating and rephrasing machine output than altering outcomes, and the promised productivity gains will show up as local time savings rather than material gains in throughput, defect reduction, customer retention, or working capital. That prediction is falsifiable, and many firms are about to make it testable. The opposite pattern will also become visible. The teams that win will not be the ones that treat AI as infallible. They will be the ones that force both human and machine output through the same causal discipline. What is the claim. What has to be true for it to work. What evidence supports it. What evidence cuts against it. What action would change if we believed it. What would we expect to observe later if the action actually shaped the outcome. That method does two things at once. It weakens ego, and it raises the learning rate. We are not trying to be right. We are trying to get it right. That sentence is easy to say and hard to live. In most organizations, being right still carries more status than getting it right. People are rewarded for sounding certain, for defending a line, and for avoiding visible reversal. Revision can look like weakness when the culture still confuses consistency with strength. But the AI era punishes that instinct. A team that cannot revise in public, based on stronger mechanism and better evidence, will turn speed into theater and caution into cost. That is why the right question for a board is no longer whether management is adopting AI. That question is already too soft. The sharper question is whether management has built a review doctrine that improves the organization’s learning rate instead of protecting internal authorship.

When a result is foreign, does the institution ask what might be true here, or does it move first to what must be wrong here. When a result feels familiar, does the institution still test the mechanism, or does fluency get mistaken for proof. The answer to those questions will tell you more about the future of the company than another demo ever will.

The gate is becoming the cost center

For years, many firms have functioned by turning experienced people into routers. They carry context across functions, explain what one system cannot tell another, win permission at each boundary, and absorb the burden of turning observation into action. That model lasted because the alternative was worse. Reasoning capacity was limited, context stayed trapped, and coordination had to be carried by human labor. That world is ending. The new pressure on the firm is not simply that machines can write, summarize, or classify. It is that they can now generate candidate reasoning paths at a scale and speed that makes human gatekeeping visible as a cost center. Not because humans are obsolete. Because humans are too expensive to keep spending their best time on work whose real purpose is preserving internal ownership of ideas rather than testing which ideas survive contact with reality. The old defense will be familiar. Leaders will say they are protecting quality. Some will be. Many will be protecting lineage. They will preserve the rule that a conclusion becomes legitimate only after it has been rewritten in the voice of the institution, signed off by the right people, and stripped of the foreignness that first made it useful. By the time that process is over, the answer may still be correct, but its option value has been spent. The window in which action could have mattered has closed. This is where Feynman’s lesson reaches past Challenger and into every operating review now underway. The issue was never merely whether management had some data. The issue was whether the institution had allowed belief, schedule, and presentation to outrank mechanism. That pattern repeats anywhere a company mistakes familiarity for validity and explanation for closure. It repeats when a recommendation is accepted because the right executive said it. It repeats when a machine recommendation is dismissed because the wrong source produced it. It repeats when teams preserve defensibility at the cost of timing and then act surprised when reality prices the delay. Reality does not care who authored the mistake. The firms that adapt will move to a harder standard. They will require machine output to be auditable enough to test, but they will also require human judgment to submit to the same discipline. They will stop treating seniority as exemption from mechanism. They will stop allowing polished language to stand in for causal proof. They will stop paying their smartest people to be customs agents for ideas crossing an internal border.

The consequence is not philosophical. It is financial. A company that keeps people as the gate will not scale judgment well enough for the volume of reasoning now available. It will produce backlog disguised as diligence. It will turn review into drag. It will keep buying intelligence while refusing to redesign the path that turns intelligence into action. That is not prudence. It is a tax on learning. The future belongs to institutions that can hold two truths at once. Models are not to be trusted because they are models. Humans are not to be trusted because they are humans. Both are to be tested against mechanism, consequence, and the record of what followed. That is the only standard that survives abundance. The reader should feel the accusation here because it is deserved. Most organizations do not suffer from a shortage of smart people. They suffer from a shortage of disciplined surrender to reality when reality threatens ownership. They want the answer, but they also want the answer to remain theirs. That desire is older than AI. AI just makes it visible. We will spend the next few years learning which firms were serious about truth and which were mainly serious about control. The difference will not show up first in speeches. It will show up in how fast the institution can convert unfamiliar but valid reasoning into defended action without first forcing it through an authorship ritual. It will show up in whether humans become the compounding mechanism for scale or remain the toll booth every answer has to clear. One posture preserves pride. The other preserves advantage.

References

This article draws on Richard Feynman’s Appendix F to the Rogers Commission report in 1986 and the commission’s own findings on the flawed Challenger launch decision, the 53 degree launch recommendation, and the temperature sensitivity of the O ring seal, because they ground the piece in a case where mechanism, timing, and filtered judgment collided in public view. It also draws on Daniel Kahneman’s work in Maps of Bounded Rationality in 2003 for the distinction between fast impression and slower checking, on Raymond Nickerson’s 1998 review of confirmation bias, on later research into belief perseverance and cognitive bias in professional decision making, and on the algorithm aversion work of Dietvorst, Simmons, and Massey in 2015, because the article’s central claim is that institutions often judge unfamiliar output more harshly than familiar output even when the familiar source is no more reliable. Research on Not Invented Here syndrome and external knowledge bias matters here for the same reason. It explains why organizations discount outside signal before they test it. The operating argument also rests on Michael Carroll’s own work on decision latency, permission architecture, auditable causal reasoning, and the principle that humans should become the compounding mechanism for scale rather than the gate.