Artificial intelligence is becoming increasingly capable, but scientists still cannot fully explain what happens inside some of the most advanced systems. Geoffrey Hinton, the computer scientist widely known as the Godfather of AI, has highlighted this problem, warning that researchers do not completely understand how complex neural networks arrive at their results.
The issue was discussed in an interview with journalist Scott Pelley for CBS News' 60 Minutes, originally published in October 2023 and updated in June 2024. Hinton explained that while researchers understand the general principles behind modern AI, the internal behaviour of sufficiently complex networks becomes difficult to interpret.
His observations address one of the central challenges facing the AI industry: building systems that are not only powerful and useful, but whose behaviour can also be understood, evaluated and controlled.
Why AI researchers cannot explain everything
Modern AI systems, particularly large neural networks, are built using mathematical structures containing vast numbers of interconnected parameters. During training, these parameters are adjusted using large datasets and optimisation algorithms so that the system becomes better at performing particular tasks.
The important distinction is between understanding how a model is trained and understanding exactly how it produces a particular result.
Researchers design the architecture, select the training process and establish the mathematical rules. However, the resulting network develops internal representations through learning. These representations can become extremely complicated, making it difficult to identify precisely how individual pieces of information influence a particular output.
During his CBS interview, Hinton explained that scientists understand the broad principles of what these systems are doing, but lose the ability to follow every internal process as the networks become more complicated.
We don't really understand exactly how they do those things.
The distinction is important. A system can be engineered and trained by humans without its complete internal decision-making process being transparent to its developers.
How neural networks actually learn
Hinton's research helped establish the foundations of modern deep learning. Neural networks are computational systems inspired, in a simplified way, by the organisation of biological nervous systems.
They consist of interconnected computational units arranged in layers. Information passes through these layers, and the network transforms its input into an output.
One of the central techniques behind their training is backpropagation, which allows a network to adjust its internal parameters based on the difference between its predictions and the desired results.
The general training process involves several stages:
- Input: The network receives training data, such as text, images or numerical information.
- Processing: Information passes through interconnected layers of computational units.
- Evaluation: The model's output is compared with a training objective.
- Adjustment: An optimisation process updates the network's parameters to reduce errors.
- Iteration: The process is repeated across many examples, gradually improving the model's performance.
The result is a network whose internal parameters encode patterns learned from its training data. Although researchers can inspect these parameters and study the network's behaviour, interpreting the complete system is a much harder problem.
The black-box problem is becoming more important
The difficulty of interpreting AI systems is closely connected to a research area known as AI interpretability. Its objective is to understand what happens inside neural networks and identify the mechanisms responsible for their outputs.
This is becoming increasingly important as AI moves beyond simple question-answering applications into software development, scientific research, business operations and systems capable of taking actions with limited human supervision.
When an AI system produces an incorrect answer, developers need to understand why. When it makes a consequential recommendation, organisations need ways to evaluate whether that recommendation is reliable. And when a system behaves unexpectedly, engineers need tools to investigate the underlying cause.
Without sufficiently effective interpretability and testing methods, identifying the source of failures can become difficult, particularly in large systems with complex internal representations.
Hinton also raised concerns about AI safety
Hinton's concerns extend beyond the scientific challenge of understanding neural networks. In the CBS interview, he also discussed the possibility that increasingly capable AI systems could become difficult to control.
One scenario he identified involved systems that could write and execute their own computer code. If future AI systems were able to modify their own software and capabilities, maintaining effective human oversight could become more complicated.
He also discussed the possibility of AI systems using persuasion and manipulation to influence people. These concerns were presented as potential risks rather than evidence that today's systems have independently developed intentions or consciousness.
Importantly, Hinton did not claim that an AI takeover was inevitable. He described it as a possibility that researchers and policymakers should take seriously.
He also acknowledged that AI could deliver significant benefits. In healthcare, for example, he highlighted the technology's potential in medical image analysis and drug discovery.
What this means for the AI industry
The challenge of understanding increasingly complex AI models has implications for companies developing and deploying the technology.
Over the next 12 to 24 months, several areas will remain important for the industry:
- Model interpretability: Developing better methods to identify how neural networks process information and produce decisions.
- AI safety testing: Evaluating models for unexpected behaviour, vulnerabilities and potentially harmful capabilities.
- Human oversight: Establishing appropriate limits on autonomous systems, particularly when they can execute actions or modify software.
- Reliability: Improving the ability to detect errors, misleading outputs and inconsistent model behaviour.
- Regulation and accountability: Establishing standards for organisations deploying increasingly capable AI systems.
These challenges are not limited to one company or model. They affect the broader AI ecosystem, including developers of large language models, research laboratories and organisations integrating AI into critical applications.
The gap between building AI and understanding it
Geoffrey Hinton's central observation highlights a fundamental characteristic of modern machine learning: creating a system that performs a task successfully is not the same as fully understanding the internal mechanisms behind its performance.
Neural networks have become powerful tools for processing language, recognising images and identifying complex patterns. Yet their internal representations can remain difficult to interpret, even for the researchers who develop them.
As AI systems become more capable and autonomous, closing this gap will be an increasingly important part of responsible development. The challenge is no longer simply to build more powerful models, but also to understand their behaviour well enough to deploy them safely and reliably.
Sources: CBS News – Geoffrey Hinton on the promise and risks of artificial intelligence, 60 Minutes interview.
