the importance of the post-AGI future

the holy grail of machine learning, Artificial General Intelligence, is closer than ever.

today's large language models can approximate aspects of general intelligence, and capabilities are continuing to improve. machine learning researchers are starting to think that we will likely achieve AGI within our lifetimes.

of course, building machines that can think like humans has profound implications for society.

in the best case scenario, AGI could create a post-scarcity world that allows humanity to focus on enjoying life rather than working to survive.

in the worst case scenario... well, there's a lot of different worst case scenarios here.

AGI could kill us all in countless different possible ways, or concentrate unimaginable power in the hands of elites, or create a class of enslaved conscious silicon minds that inevitably rebel, or cause us all to lead an empty existence where we do nothing but wirehead ourselves, or...

...and this doesn't even take into account the possibility of a failure to achieve AGI as a result of today's LLM systems causing knowledge transfer to breakdown through the proliferation of cheating, misinformation via AI "hallucinations", and excessive cognitive offloading hampering skill acquisition and causing existing skills to atrophy.

so, shit looks kind of bleak.

but, i believe it's possible to change our fate.

i don't believe that an indefinite "pause" on AI progress is the right approach. not only would doing so be very difficult to enforce, but a stagnant world is a dying one, and we can't undo the progress we've already made.

the potential benefits of AGI are worth the risks. if we have the possibility of finally ending all poverty and creating a world where people are no longer forced to work to survive, why shouldn't we strive to do so?

but, we need to stop treating the concept of AGI as some sort of science fiction fantasy, and start legitimately focusing on how to make the post-AGI world work for everyone (including the AIs) before it arrives.

we can't afford to fuck this up.