Key Takeaways
- In a study from researchers at three French and Italian universities, access to AI advice was linked to a drop in accuracy and a sharp fall in people saying they did not know the answer.
- Confidence went up even as performance went down, suggesting the tool made some people more willing to trust incorrect responses.
- The researchers say the pattern points to a broader problem: AI systems may discourage the habit of recognizing uncertainty.
What happened
Researchers from three universities in France and Italy studied how access to AI advice affected people answering questions that the model was not especially good at. The questions included visual details from films, such as the color of a team’s uniform in Bend It Like Beckham.
According to the study summary, the presence of AI advice changed how people answered in three important ways. The share of participants willing to suspend judgment and say “I don’t know” fell from 44% to 3%. Accuracy dropped from 27% to 9%. At the same time, confidence increased from 30% to 76%.
Valerio Capraro, an associate professor at the University of Milano-Bicocca and one of the authors, said the effect was not small. “People became much worse, the accuracy was only one third, but they were twice as confident,” he said.
The researchers deliberately used a model that was usually wrong on these questions, Step 3.5 Flash, so the results would not be explained by sensible reliance on a reliable assistant. In some cases, participants who would have answered correctly on their own asked the AI and then changed to a wrong answer.
The study also tested whether incentives changed behavior. Monetary rewards helped somewhat, but the underlying pattern remained. Willingness to admit ignorance rose from 3% to 8%, and accuracy improved from 9% to 16%, but both stayed below the no-AI baselines.
The source also connects the findings to earlier work from Wharton researchers, who used the term “cognitive surrender” to describe people accepting incorrect AI answers while feeling more confident than those working without AI.
Why it matters
The central technology question here is not whether AI can answer questions, but how its presence changes human judgment. The study suggests that AI advice can suppress a basic critical-thinking habit: knowing when you do not know enough to answer.
That matters because many consumer and workplace AI products are designed to provide a response, not to pause or express uncertainty. If users start treating confident-looking outputs as substitutes for their own judgment, errors can spread even when the AI is wrong.
Capraro said he is especially concerned about children, who are growing up with these systems before they have fully developed critical thinking skills. The source also points to current debates around AI search and school use, including Common Sense Media’s warning about Google’s AI search summaries. The broader concern is that interface design may train users to expect certainty where uncertainty would be more appropriate.
What to watch
One thing to watch is whether similar effects show up in other settings beyond the study’s film-related questions. The source makes clear that the researchers chose questions where the model tended to fail, so the result is about overreliance on a weak AI assistant rather than delegation to a strong one.
It will also be important to see whether future AI products add clearer uncertainty signals, better refusal behavior, or stronger guardrails around low-confidence answers. The source implies that current systems often answer even when they should not.
A final question is how education, workplace policy, and product design respond to the same behavioral pattern: people may become less willing to say “I don’t know” when AI is available. If that pattern holds, the issue is not just model accuracy. It is the way AI changes how people think about their own knowledge.
