Pressure to adopt AI now comes from two directions: boards want to see an AI strategy acted on, and employees are already using AI at work (whether leaders know it or not). The instinct is to invest in training, increasing AI literacy and getting everyone on the same page. The assumption underneath is that if people know more about AI, they will use it well.

But what happens when those training programmes aren't having the impact you'd expect?

A study published earlier this year in Computers in Human Behavior (Fernandes et al., 2026) tested what happens when people use AI to solve complex reasoning problems. Across roughly 700 participants in two studies, AI improved task performance (so far, so good). People genuinely did better work, but they also systematically overestimated how well they had performed. They thought they were doing even better than they were.

You could argue no harm, no foul. The work improved, and that is what matters. But overconfidence in AI-assisted output has a cost that shows up downstream. If you cannot accurately assess how good your AI-assisted work actually is, you lose the ability to know when it falls short. You stop scrutinising the output, and decisions get made on the back of analysis that everyone involved believes was better than it actually was. In professional services, the gap between excellent and adequate is where liability, reputation, and client trust live.

And there is a more unsettling finding. Participants with higher AI literacy (that is, more technical knowledge of how AI works) were less accurate in assessing their own performance, not more. Knowing more about AI made people more confident in their AI-assisted results. But that confidence was misplaced.

The standard intervention, teaching people more about how AI works, may actively undermine the judgment it is supposed to build.

What effective users actually do differently

The Fernandes finding points to something that cognitive scientists have studied for decades: metacognition, or the ability to monitor and evaluate your own thinking. Research on metacognition consistently shows it is one of the strongest predictors of whether someone uses tools and information effectively, or just confidently. In the context of AI, the question is whether users can distinguish between "AI helped me do better work" and "AI made me feel like I did better work." The Fernandes data suggests most cannot.

But a small percentage of users do something fundamentally different. The NeuroLeadership Institute (NLI), observing how people use AI across organisations, describes a pattern. The most effective users do not ask AI for answers to complex problems. They arrive with their own position, use AI to pressure-test it, and maintain what NLI calls "intellectual authority" throughout. The distinguishing habit is not technical fluency but the practice of thinking about your own thinking. Staying aware of what you know, what you do not know, and where AI's confident output might be masking a gap.

The difference is visible in how people respond to AI output. A partner reviewing a junior's rough draft reads with a red pen. The same partner reviewing an AI-generated memo that arrives polished, formatted, and confidently structured reads with a different mindset. The surface signals say "this does not need my scrutiny." But it does.

Metacognition is not an innate talent. It is a trainable cognitive habit. But almost no AI training programme is designed to build it.

What this means

These are independent research teams, studying different populations, using different methods. They reach the same conclusion. Proficient use of AI is a learnable skill, but what needs to be learned is not what most organisations are teaching.

The important (and teachable) skill is not prompt engineering, not tool fluency, not AI literacy. It is evaluative judgment, the ability to know what good looks like in your domain and to maintain that standard when AI produces something that looks polished but may not be sound.

Part of that judgment is knowing when not to use AI at all. Other research challenges the "use AI on everything" narrative: AI helps most where the human's existing capability is lowest, and helps least (sometimes hurts) where the human is already operating at a high level. A large-scale study of customer support workers (Brynjolfsson et al., 2025, Quarterly Journal of Economics) found that AI compressed the performance distribution, lifting less experienced workers while the highest performers saw small declines in quality.

A randomised controlled trial by METR, a nonprofit AI research organisation, found the same pattern in sharper relief. AI slowed experienced software developers down by 19% on their own codebases. They believed it had made them 20% faster. The gap between what they perceived and what actually happened was 39 percentage points (note that this was pre-Opus 4.6).

The people most organisations assume need the least help with AI - their most experienced practitioners, their senior partners - are the ones the research suggests are most vulnerable to misjudging it. Not because they lack skill, but because their expertise makes them confident in evaluations that may no longer be reliable when AI is producing the work. They know what good looks like, and AI is increasingly good at looking like it.

This does not mean experienced practitioners should avoid AI. It means the skill includes knowing where AI earns its place and where it does not. For a senior practitioner, that is probably not the core judgement-heavy work they already do at peak quality. It is more likely the operational tasks around that core work, the research in areas adjacent to their specialty, or the first-pass synthesis their team would otherwise bring to them for review. The evaluation skill is not just "assess what AI produces," it is "decide what to give AI in the first place."

For leaders running organisations, the implication is practical. The skill that makes someone effective with AI is the same skill that makes someone a good professional practitioner. It is the ability to evaluate work against a standard, to notice when something looks right but is not, to hold judgment when confidence is easier. For a large firm, that capacity is built through supervision, case review, mentoring, and reflective practice. For a smaller firm or a sole practitioner, it is the expertise the owner carries and the professional standard they apply to every piece of work, whether a junior produced it or an AI did. Either way, the infrastructure exists. It just has not been connected to AI.

When I analysed how I use AI across several months of work, the pattern that emerged was not about prompting. The investment that made AI most useful was arriving with clear criteria for what good looks like, and holding that standard when the output came back looking confident and complete. The moments where AI was least useful were the moments where I had no clear picture of what I was evaluating against. The effective users NLI observed do the same thing. Fernandes's overconfident participants did not.

The approach I take to building AI capability looks counterintuitive from the outside. I do not start with what AI can do. I start with what people already know good work looks like. Then we test AI against that standard. The skill that transfers is not prompting. It is the habit of predicting what you think AI will produce, comparing it to what good actually looks like, and paying attention to where your confidence was wrong.

We do not yet have a study proving this at scale, but the evidence suggests the right kind of structured AI practice builds the reflex. The alternative (more tool training, more literacy programmes) is the one approach the research suggests makes the problem worse. Most AI training builds confidence. What leaders actually need is calibrated judgement.\

Sources

Fernandes et al., AI Makes You Smarter but None the Wiser, Computers in Human Behavior (2026, roughly 700 participants across two studies)

Brynjolfsson, Li & Raymond, Generative AI at Work, Quarterly Journal of Economics (2025, 5,172 customer support agents)

METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025, 16 developers, 246 tasks)

Rock & Weller, The One Skill That Separates People Who Get Smarter with AI from Everyone Else, Fortune (March 2026, based on NeuroLeadership Institute research)