Eliezer Yudkowsky
Director and Research Fellow, The Singularity Institute
A seed AI is a strongly self–improving process, characterized by improvements to the content base that exert direct positive feedback on the intelligence of the underlying improving process. The exponential surge of human cultural knowledge was driven by the action of an already–powerful but constant force, human intelligence, upon a synergetic content base of cultural knowledge. Since strong self–improvement in seed AI involves an initially very weak but improving intelligence, it is not possible to conclude from analogies with human cultural progress that strongly recursive self–improvement will obey an exponential lower bound during early stages, nor that it will obey an exponential upper bound during later stages. Strong self–improvement is a mixed blessing in development. During earlier epochs of seed AI, the dual process of programmer improvement and self–improvement probably sums to a process entirely dominated by the human programmers. We cannot rely on exponential bootstrapping from an unintelligent core. However, we may be able to achieve powerful results by bootstrapping from an intelligent core, if and when such a core is achieved. Recursive self–improvement is a consequence of seed AI, not a cheap way to achieve AI.
Biography
Eliezer Yudkowsky is Research Fellow and Director of the Singularity Institute for Artificial Intelligence, a non–profit research institute dedicated to increasing the likelihood of, and decreasing the time to, a maximally beneficial singularity. He is one of the world's foremost experts on the subject, and frequently speaks on artificial general intelligence (AGI), rationality, and the future.
Yudkowsky created the “Friendly AI” approach to AGI, which emphasizes the importance of the structure of an ethical optimization process and its supergoal, in contrast to the more common trend of seeking the 'right' fixed enumeration of ethical rules a moral agent should follow. In 1996, he wrote “Staring into the Singularity,” one of the first singularity analyses. In 2001, he presented the first technical design for a self–improving AI that takes safety into consideration, with his book–length Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures. “Coherent Extrapolated Volition” (2004) is the most recent written update to his Friendly AI theory.
Yudkowsky's research into the workings of the human mind and minds–in–general led to a paper on the evolutionary psychology of human general intelligence, “Levels of Organization in General Intelligence” (2002), forthcoming in a volume on Artificial General Intelligence, as well as papers on human rationality and philosophy of science, including “An Intuitive Explanation of Bayesian Reasoning,” “A Technical Explanation of Technical Explanation,” and “The Simple Truth.” He also has two papers forthcoming in an edited volume on Global Catastrophic Risks from Oxford University Press, entitled “Cognitive Biases Potentially Affecting Judgment of Global Risks” and “Artificial Intelligence and Global Risk.”