AGI is coming, and we can't keep pretending that it's not

The first step to solving a problem is recognizing there is one.

there's this common view in left-leaning circles that all the developments going on in generative AI are nothing more than a big grift. the idea is that we're burning enormous sums of money on an endeavour that will never bear any fruit besides burning the commons to the ground and making a few people rich.

well, they're half-right here.

we may possibly be in a generative AI bubble, as many of the leading AI companies are spending at an unsustainable rate and have valuations based upon unrealistically optimistic expectations.

and, generative AI is filling the commons with slop that is becoming indistinguishable from reality, destroying the incentives to create human-written online content, making academic dishonesty easier than ever, and weakening our ability to think for ourselves.

and yet, i don't believe it's all a grift.
i believe there's something more here besides just burning the commons to the ground in order to make a bit of money.

and i believe what's coming next is terrifying.

science-fiction has always talked about the risks of future AI systems with cognitive abilities comparable to or exceeding that of humans. in the field of machine learning, this hypothetical future AI system is referred to as Artificial General Intelligence (AGI).

and despite all the limitations of our current generative AI systems, they've shown us that AGI isn't as far away as we thought.

early on in LLM research, researchers began to notice that these systems developed unexpected emergent abilities as they were improved, and eventually these emergent abilities became so advanced that the parallels between LLM systems and AGI became too obvious to ignore.

but of course, these systems aren't AGI. they're by far the closest we've ever been to AGI, but they still frequently make weird mistakes that no human would make, limiting their usefulness in the real world.

so, how the hell do we even determine what is and isn't AGI? prior to LLMs, it was a "you know it when you see it" kind of thing, but now we have something that behaves kind of like AGI but clearly isn't, so we really need a formal definition.

it turns out that doing this is really hard.

as LLMs get better and better, they end up saturating our benchmarks of their abilities. we're now throwing the hardest academic tests we possibly can at them (which encompass such a large variety of Ph.D.-level questions that it's unlikely that any one human could beat them), and LLMs are still rapidly getting better at them.

yet, despite these amazing accomplishments, they still fail the "you know it when you see it" test. maybe academic challenges are the wrong approach?

there has been a recent focus on using abstract logical reasoning puzzles to test LLMs instead, and again, LLMs are rapidly getting better at them, forcing the creation of more advanced logical reasoning tests in order to find challenges where LLMs can't yet achieve human-level performance.

for a while, a lot of the people i talked to on twitter used an outcome-based definition of AGI: AGI is an AI system that is capable of performing all intellectual labor currently performed by humans.
however, i'd argue that this isn't that much more useful than the "you know it when you see it" test, as it doesn't provide meaningful guidance on how to improve these systems.

the most recent attempt at differentiating LLMs from AGI has been a comprehensive suite of tests based on human psychometrics research, and it does a good job at showing where current systems fall short:

• they can't truly learn new things after their initial training ("memory" systems in current LLMs are fairly limited hacks)
• their emergent abilities don't yet generalize to non-textual inputs
• they still struggle with certain tasks that humans find trivial (likely because they perceive words differently than humans do?)
• they struggle with tasks that require complex multi-step reasoning (but they are consistently getting better and better at this)

anyways, the launch of ChatGPT has sparked an ongoing arms race to build AGI. LLMs are still continuing to get better and better, and according to a recent survey of AI researchers, we will likely achieve AGI within our lifetimes.

however, the problem is that we have no clue what the hell we're going to actually do once we achieve AGI.

most science-fiction focuses on the risk of a misaligned AI leading to the extinction of humanity. this is absolutely a valid concern (for example: we seriously struggle to align our current AI systems, and even a 1% risk of human extinction is too great), but i worry that this focus on misaligned AGI's existential risk causes us to overlook the societal impact of "aligned" AGI.

AGI will break the social contract that our society is built upon by removing the incentives for those in positions of power to care about ordinary people. the reason that people in positions of power care about you is because you are able to create economic value for them, and once AGI exists, it's only a matter of time until it is able to outcompete you economically.

in our pre-AGI society, you also affect their ability to hold onto power, either due to your ability to participate in democracy or by the threat of a political movement deciding to overthrow the government. however, AGI will change the balance of power, making coups much easier for those in power to pull off and preventing armed uprisings from being able to threaten the state.

even if strong institutions are somehow able to successfully prevent internal power struggles and universal basic income gets implemented, a perceived lack of upward mobility could still lead to political instability.

and even if AGI somehow doesn't kill us all, lead to extreme inequality that ends up killing almost all of us, or result in us all living in hellish authoritarian surveillance states, it's going to create a crisis of meaning. what's the point of doing anything when superintelligent machines can do everything much better than you can?

we need to start preparing for the realities of a post-AGI world now, rather than treating like it's a science fiction fantasy until it's too late.

i wish i had all the answers and could say exactly what we need to do to ensure a good outcome. i don't; nobody does.

the best time to have started thinking about how to mitigate the harmful effects of AGI was a decade ago. the second best time is right now.