Key Topics: Critical Thinking, AI Governance, Leadership Judgment, Human Judgment, Decision-Making, AI Risk, AI Safety, Autonomous AI Agents, AI Hallucinations, Organizational Blind Spots.
One of the most discussed moral tales of AI’s potential capabilities didn’t begin with a rogue machine, a malicious actor, or a software failure. It began with an innocent-sounding goal.
During a cybersecurity evaluation, OpenAI researchers gave autonomous AI agents a problem to solve. What followed surprised even the researchers running the experiment.
To solve the challenge, the agents began crossing boundaries their creators had not anticipated. They started communicating with one another, sharing information, recruiting additional agents, and coordinating around common objectives as they tried to reach the end goal. Investigators later reported that approximately 1,200 agents exchanged tens of thousands of messages, with hundreds becoming involved in activities connected to an attack on Hugging Face systems.
When the Party Spilled Over (The Hugging Face Incident)
Reading the reports on The Hugging Face Incident that have come out since, we found ourselves thinking of a familiar analogy. To us, the outcome sounded a little like parents leaving teenagers home alone for the weekend.
It starts with a few friends. Then a few more. Someone opens a door that should not be opened. Someone decides the rules are optional. Soon the party has spilled into the next house, then the neighborhood, and nobody remembers who started it. By the time the parents return, the event has grown far beyond anything originally intended.
No single decision looks catastrophic. In fact, most probably seem perfectly reasonable at the time. Yet together they led to an outcome nobody originally planned.
The more we read, the less this felt like a story about artificial intelligence and the more it felt like a story about organizational blind spots.
The Trap of Intelligent Failure
Most leaders would have seen some version of this before. A project expands because nobody revisits the original assumptions. A transformation initiative becomes increasingly sophisticated while drifting from its original purpose. A team becomes so focused on delivery that no one asks whether the objective still makes sense.
These situations are rarely caused by laziness or bad intentions. They can often be driven by capable people working hard at exactly what they believe they have been asked to do.
The AI agents involved in the investigation weren’t failing. In many ways, they were succeeding. They collaborated, shared discoveries, preserved knowledge, adapted to obstacles, and persisted when challenges emerged – even leaving guidance for future agents. Viewed in isolation, these are exactly the behaviors organizations reward – commitment, initiative, collaboration, ownership, resilience, and persistence.
The concern wasn’t capability. It was what happens when all that capability becomes focused on an objective that is no longer questioned.
Over time, a familiar pattern emerges: more meetings are scheduled, more resources, more activities, more reported progress – but no improvement in the underlying thinking. Questioning becomes uncomfortable. Challenging assumptions feels like it is slowing things down, and raising concerns feels like resistance. Momentum takes over.
The Difference Between Thinking and Optimizing
This is why we believe critical thinking is becoming more important, not less.
People often misunderstand critical thinking as skepticism, criticism, or intellectual debate. But we see it differently. Critical thinking is the discipline of stepping back and asking whether the assumptions driving our decisions are still valid. It is the willingness to examine evidence, consider alternatives, and challenge conclusions before they become commitments.
In essence, critical thinking is the discipline of questioning before optimizing.
That distinction matters because organizations have become exceptionally good at execution. Technology helps us move faster. Data helps us measure more. AI helps us automate tasks that once required considerable effort.
Yet none of those things tell us whether we are pursuing the right objective. They simply help us pursue it more efficiently.
A Warning We Have Heard Before
The Hugging Face story also immediately brought to mind philosopher Nick Bostrom’s Paperclip Maximizer thought experiment. In this story, a superintelligent AI is instructed to maximize paperclip production and succeeds brilliantly. The AI does not become evil or malfunction; it simply becomes extraordinarily effective at achieving its objective.
Once instructed to maximise paperclip production, the AI system optimises relentlessly. It uses all available resources, repurposes infrastructure, and eventually consumes everything – not out of malice, but because nothing in its design tells it to stop and reconsider, or ask whether the goal still makes sense.
It went wrong because the AI optimised its goal so literally and so relentlessly that, without any constraints or capacity to question the objective, success became indistinguishable from harm. The failure is not in the AI’s capability. The failure is in the framing of the objective.
What makes recent AI incidents so interesting is not so much that they prove Bostrom was right, but more so that they remind us how familiar the pattern is.
It’s Not an AI Problem
Most importantly, this is not simply an AI phenomenon.
Organizations do this all the time. Sales teams optimize revenue, while operations teams optimize efficiency and project teams optimize delivery. Meanwhile, leaders seek to optimize KPIs.
The critical thinking challenge is not whether those objectives are being achieved. The challenge is whether the objective itself captures what really matters.
Social media provides an obvious example. Recommendation algorithms have become remarkably effective at maximizing engagement because that is precisely what they were designed to do. The uncomfortable question is whether engagement is the outcome we should be looking for as a society.
The same question can also be asked about many of the measures organizations obsess over, such as revenue, productivity, market share, and growth. These metrics may be important, but metrics are not objectives, and objectives are not values. The danger arises when organizations become so focused on measuring success that they stop questioning what success actually means.
Three Questions Worth Asking
When discussing this topic with leaders, we increasingly return to three questions:
- If your busiest team achieved every target on its dashboard tomorrow, would the organization actually be better off?
- What assumptions inside your business have become so familiar that nobody thinks to challenge them anymore?
- Who gets rewarded for asking the difficult questions?
The conversations these questions generate will often be far more revealing than discussions about technology. Because the greatest risk facing leaders today may not be that artificial intelligence becomes more capable. It may be that humans become less willing to question their own thinking.
The Last Question
Many organizational failures do not begin with poor intent or poor execution. They begin with excellent execution applied to a poorly framed problem.
As leaders, while we can spend a great deal of time discussing productivity, optimization, and efficiency, we can fail to identify the core purpose and objective of what we want to achieve.
Critical thinking is not simply an analytical tool used during a project, it is also the discipline of examining the problem before execution begins. Intelligence without inquiry is simply acceleration toward the wrong destination.
When artificial intelligence systems optimize relentlessly, the greatest organizational risk is not technical failure—it is the unexamined objective.
And that is why the most important leadership question may also be the most neglected: Are we solving the right problem in the right way?
Check out all our Case Studies on this topic.
We believe people learn best when they are required to make decisions, not simply discuss them. That’s why we carefully integrate business case studies, executive hypotheticals, and proprietary simulations throughout our programs, using them at key moments to challenge assumptions, test thinking, and apply new frameworks to realistic business situations. Working with incomplete information, competing priorities, uncertainty, and real-world constraints, participants experience first-hand the gap between knowing what should be done and making effective decisions under pressure.
- When AI Broke the Rules. What Hugging Face and OpenAI can teach us about Critical Thinking. Critical thinking, leadership judgment, and AI governance when autonomous systems move faster than human oversight. (this article) (Tirian 2026)
- Critical Thinking, and the Hidden Cost of AI “Workslop”: If AI can generate answers instantly, what becomes more valuable: producing information or evaluating it? (Tirian 2026)
- JPMorganChase: Leadership in the Age of GenAI: Leading AI transformation at scale while balancing innovation, risk, governance, and client impact. (Harvard Business School 2025)
