Feature: Humans have always been black boxes

IMAGINE ASKING A COLLEAGUE why they made a critical call. They pause, think carefully, and say: ‘I’m not entirely sure. It just felt right.’

We accept this. We nod. We move on.

Now imagine an AI system making the same decision, and when asked to explain itself, producing the same answer: it processed the data, something happened in
the middle, and an output emerged. No clear rationale.
No traceable reasoning. Just a result.

We find this unacceptable.

But here is the uncomfortable truth that sits at the intersection of neuroscience, leadership, and artificial intelligence: the two scenarios are not as different as
we’d like to believe.

Humans have always been black boxes. We just
never called it that.

WHAT THE AI PROBLEM ACTUALLY IS

In machine learning, the ‘black box’ problem refers to the fact that the internal workings of complex AI systems are largely opaque. Data enters. An output emerges. But the billions of calculations that happened in between? Even the engineers who built the system cannot fully explain them.

Here is the problem: when an algorithm decides, someone’s life could be affected by a decision no one can fully explain. That is not good enough. So an entire field, Explainable AI, or XAI, now exists to do one thing: open the box. Not to simplify what is complex but to trace the reasoning.

THE BOX WE CARRY

The human brain contains roughly 86 billion neurons. Every experience you have ever had, every message absorbed about your worth, your capability, your place in the room. None of it disappeared; it was absorbed, encoded, and built into the very architecture of how you think and decide. And despite everything neuroscience now knows about the brain, it still cannot fully trace how that shaping works.

But here is what neuroscience is clear about: the vast majority of decisions we make, including professional, strategic, high-stakes ones, are generated beneath conscious awareness. Your brain comes to a conclusion and then quickly provides your conscious mind with a justification. This justification feels true and appears well thought out. Psychologists refer to this process as confabulation, essentially, the honest lie that the mind tells itself. It’s not a form of manipulation; rather, it’s simply the mind functioning as it was designed to do: making sense of its thoughts after the fact.

Data in. Decision out. The middle? A mystery, even to the person making the call.

We built AI in our own image and then we’re surprised that it has the same problem we do.

THE COST OF UNEXPLAINED DECISIONS

In the AI context, ‘unexplainability’ creates measurable risk. Bias goes undetected. Trust erodes. Accountability disappears. When something goes wrong, no one can trace it to its source or fix it.

Leadership carries the same risk, at the same scale.

The senior leader who consistently avoids certain conversations but cannot explain why. The decision-maker who repeatedly undervalues a particular type of contribution without recognising the pattern. The high-performer who diminishes in situations where they should assert authority, despite every external credential suggesting otherwise.

These are not character flaws. They are the outputs of an unexamined system, a black box running at full capacity, shaping decisions and influencing culture without anyone, including the person themselves, understanding the inputs that produced them.

The behaviours of leaders do not stay with them. They spread, through the team, through what gets said and what remains permanently unsaid. When leadership operates from unexamined patterns, the organisation inherits the variance.

THE TOOLS WE’VE BUILT,
IN BOTH WORLDS

The AI community is investing heavily in interpretability: making the system legible enough to be trusted, corrected, and improved. Humans have been building equivalent tools for thousands of years: therapy, philosophy, coaching, and reflective practice. All of them, at their core, ask the same question: what actually produced that output?

We just called them different things. And in professional contexts, we have often treated them as optional, the domain of personal life rather than leadership performance.

Self-awareness is not a soft skill. It is the interpretability layer of human leadership.

THE QUESTION WORTH SITTING WITH

We are in a period of significant examination of artificial intelligence, including its decisions, biases, and accountability. This examination is both justified and essential. But it raises a question we apply far less rigorously to ourselves:

If we cannot yet fully explain ourselves, our patterns, our triggers, our unexamined assumptions, what does that mean for the quality of leadership we are currently providing?

The leaders who will shape the next decade are not simply those with the sharpest analytical minds. They are those who have done the harder, less visible work: becoming legible to themselves. Because in a world of increasing complexity and accountability, the most dangerous leader in any room is the one whose decision-making is a black box, to everyone, including themselves.

Author

Usha Maharaj CA(SA), SAICA Chairman’s Awards 2025 Winner for the Wellness Advocate Category, Leadership Facilitator, Speaker, Coach, hello@ushamaharaj.com