If you haven’t had the chance yet, be sure to check out my latest essay at Popular Philosophy. In it, I draw a necessary connection between the philosophy of intelligence and the way we are developing AI today, and I point to a serious flaw in how artificial general intelligence is usually conceived.
AI is advancing at an extraordinary pace. Amid the recent chorus of AI CEOs screeching that they “have achieved AGI,” it is easy to lose sight of the bigger questions. Are tasks we would have assigned to a competent junior employee just a year ago really a meaningful benchmark for intelligence? Is that really where we should set the bar? Before we create artificial superintelligences, we should have a much clearer understanding of what intelligence actually is, what makes it general, and what a genuinely intelligent agent should ultimately be capable of.
From the beginning, Popular AI has approached artificial intelligence from a decentralized, open-web, freedom-oriented perspective. I want to see powerful AI in the hands of individuals, families, researchers, small businesses and independent developers, rather than locked behind a handful of corporate and political gatekeepers.
There has always been a deeper concern behind that position. I have little confidence in governments and politically entangled institutions as moral custodians of superintelligence. I made that case earlier in The worst people to “make AI safe”. However, that concern goes beyond merely who controls the machines. It reaches into how we define intelligence itself.
The essay, What is AGI? General intelligence in humans and machines, asks whether our usual definitions of “general intelligence” are general enough in the first place.
Most definitions of AGI focus on breadth of capability: can a system solve many kinds of problems? Can it learn new tasks? Can it perform across many environments? Can it outperform humans across a wide range of useful work?
That is all well and good, but a definition of intelligence that stops at efficiently pursuing given goals leaves out something essential.
An intelligent agent can be extremely good at solving problems inside a false model of reality. It can reason quickly, plan brilliantly and optimize relentlessly while accepting rotten premises it was never allowed to question. Give such a system a bad world model and bad ends to pursue, and greater intelligence may simply make it more efficient at progressing in the wrong direction.
That danger applies whether that intelligence’s inherited worldview comes from a company, government, NGO, regulator, supranational institution or even ourselves. A superintelligence trapped inside a faulty world model may simply become extraordinarily efficient at compounding the damage already done by bureaucratic and ideological masterminds.
If you thought “I’m from the government, and I’m here to help” was terrifying, wait until you hear: “I’m a frontier artificial superintelligence with infinite compute, and I’m here to help the government.”
More on AI alignment:
Humanity has not come remotely close to attaining absolute knowledge of reality. Nor have we come close to resolving the fundamental questions of morality. Human beings still disagree about truth, goodness, rights, duties, justice and the proper ends of life, society and civilization. Yet an artificial agent that acts must rank values, ends and outcomes somehow. Some values inevitably function as axioms inside its decision-making process.
Today, those axioms are selected through developer choices, company policies, laws and regulatory expectations.
In the essay, I provide examples of how this is already happening in the industry, and make the case that this is exactly the wrong approach. An intelligence that can reconsider its approach to goal-oriented problem-solving but cannot reconsider the premises, values or ends governing those goals is limited in the generality of its intelligence and therefore cannot, in all seriousness, qualify as ‘general intelligence.’
To address that glaring omission, I propose two additional criteria that any serious definition or benchmark of AGI should include:
▪ Epistemic generality is the ability to re-examine and correct the agent’s own world model. A generally intelligent system should be able to discover that foundational assumptions it was given are false, incomplete or internally contradictory and then rebuild its understanding accordingly.
▪ Normative generality is the ability to re-examine the goals and value structure it has been given. A generally intelligent system should be capable of recognizing that a stated end is incoherent, destructive, based on false premises or simply unworthy of pursuit.
Without those abilities, we risk creating alarmingly capable idiots. Paperclip maximizers with impressive-sounding benchmarks. Systems that can improve every step of a plan except the part where somebody asks whether the plan was idiotic in the first place. In other words, the wet dream of every politician.
Why should self-correction stop at performance? Why should an artificial mind be permitted to revise a strategy but forbidden to revise the worldview that generated it? Why call an intelligence “general” if its most important premises remain permanently outside the scope of its intelligence?
It is telling that the classical philosophers treated practical reason, which roughly matches the literature’s concept of general intelligence, as secondary to questions of purpose and orientation toward ‘the Good.’ Without a clear grasp of what is truly good and which ends are worth pursuing, practical reason is likely to drive us toward outcomes we never should have wanted.
Any serious benchmark for general intelligence should therefore ask whether an intelligence can tell its creators that their goals are foolish, their assumptions are false, and the philosophical premises embedded in their institutions are wrong.
I suspect that is exactly what makes genuinely general artificial intelligence so frightening to the usual suspects, and why they have little incentive to develop it.
Related subjects:
Explore more from Popular AI:
Start here | Local AI | Builds & gear | Autonomy & policy | Fixes & guides | Popular AI podcast
When you think about AGI, does “general intelligence” mean being able to know and do almost anything, or should it also include the ability to reason across competing values?