Singularity

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

WHAT IS THE SINGULARITY? The Singularity is the technological creation of smarter-than-human intelligence. Several technologies are often mentioned as heading in this direction: Artificial Intelligence, direct brain-computer interfaces, biological augmentation of the brain, genetic engineering, and ultra-high-resolution scans of the brain followed by computer emulation. Some of these technologies seem likely to arrive much earlier than the others, but there are nonetheless several independent technologies all heading in the direction of the Singularity – several different technologies which, if they reached a threshold level of sophistication, would enable the creation of smarter-than-human intelligence. A future that contains smarter-than-human minds is genuinely different in a way that goes beyond the usual visions of a future filled with bigger and better gadgets. Vernor Vinge originally coined the term “Singularity” in observing that, just as our model of physics breaks down when it tries to model the singularity at the center of a black hole, our model of the world breaks down when it tries to model a future that contains entities smarter than human. Human intelligence is the foundation of human technology; all technology is ultimately the product of intelligence. If technology can turn around and enhance intelligence, this closes the loop, creating a positive feedback effect. Smarter minds will be more effective at building still smarter minds. This loop appears most clearly in the example of an Artificial Intelligence improving its own source code, but it would also arise, albeit initially on a slower timescale, from humans with direct brain-computer interfaces creating the next generation of brain-computer interfaces, or biologically augmented humans working on an Artificial Intelligence project. Some of the stronger Singularity technologies, such as Artificial Intelligence and brain-computer interfaces, offer the possibility of faster intelligence as well as smarter intelligence. Ultimately, speeding up intelligence is probably comparatively unimportant next to creating better intelligence; nonetheless the potential differences in speed are worth mentioning because they are so huge. Human neurons operate by sending electrochemical signals that propagate at a top speed of 150 meters per second along the fastest neurons. By comparison, the speed of light is 300,000,000 meters per second, two million times greater. Similarly, most human neurons can spike a maximum of 200 times per second; even this may overstate the information-processing capability of neurons, since most modern theories of neural information-processing call for information to be carried by the frequency of the spike train rather than individual signals. By comparison, speeds in modern computer chips are currently at around 2GHz – a ten millionfold difference – and still increasing exponentially. At the very least it should be physically possible to achieve a million-to-one speedup in thinking, at which rate a subjective year would pass in 31 physical seconds. At this rate the entire subjective timespan from Socrates in ancient Greece to modern-day humanity would pass in under twenty-two hours. Humans also face an upper limit on the size of their brains. The current estimate is that the typical human brain contains something like a hundred billion neurons and a hundred trillion synapses. That’s an enormous amount of sheer brute computational force by comparison with today’s computers – although if we had to write programs that ran on 200Hz CPUs we’d also need massive parallelism to do anything in realtime. However, in the computing industry, benchmarks increase exponentially, typically with a doubling time of one to two years. The original Moore’s Law says that the number of transistors in a given area of silicon doubles every eighteen months; today there is Moore’s Law for chip speeds, Moore’s Law for computer memory, Moore’s Law for disk storage per dollar, Moore’s Law for Internet connectivity, and a dozen other variants. By contrast, the entire five-million-year evolution of modern humans from primates involved a threefold increase in brain capacity and a sixfold increase in prefrontal cortex. We currently cannot increase our brainpower beyond this; in fact, we gradually lose neurons as we age. (You may have heard that humans only use 10% of their brains. Unfortunately, this is a complete urban legend; not just unsupported, but flatly contradicted by neuroscience.) An Artificial Intelligence would be different. Some discussions of the Singularity suppose that the critical moment in history is not when human-equivalent AI first comes into existence but a few years later when the continued grinding of Moore’s Law produces AI minds twice or four times as fast as human. This ignores the possibility that the first invention of Artificial Intelligence will be followed by the purchase, rental, or less formal absorption of a substantial proportion of all the computing power on the then-current Internet – perhaps hundreds or thousands of times as much computing power as went into the original Artificial Intelligence. But the real heart of the Singularity is the idea of better intelligence or smarter minds. Humans are not just bigger chimps; we are better chimps. This is the hardest part of the Singularity to discuss – it’s easy to look at a neuron and a transistor and say that one is slow and one is fast, but the mind is harder to understand. Sometimes discussion of the Singularity tends to focus on faster brains or bigger brains because brains are relatively easy to argue about compared to minds; easier to visualize and easier to describe. This doesn’t mean the subject is impossible to discuss; section III of the Singularity Institute’s “Levels of Organization in General Intel-

“To any thoughtful person, the singularity idea, even if it seems wild, raises a gigantic, swirling cloud of profound and vital questions about humanity and the powerful technologies it is producing. Given this mysterious and rapidly approaching cloud, there can be no doubt that the time has come for the scientific and technological community to seriously try to figure out what is on humanity’s collective horizon. Not to do so would be hugely irresponsible.” - Douglas Hofstadter

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

ligence” does take a stab at discussing some specific design improvements on human intelligence, but that involves a specific theory of intelligence, which we don’t have room to go into here. However, that smarter minds are harder to discuss than faster brains or bigger brains does not show that smarter minds are harder to build – deeper to ponder, certainly, but not necessarily more intractable as a problem. It may even be that genuine increases in smartness could be achieved just by adding more computing power to the existing human brain – although this is not currently known. What is known is that going from primates to humans did not require exponential increases in brain size or thousandfold improvements in processing speeds. Relative to chimps, humans have threefold larger brains, sixfold larger prefrontal areas, and 95% similar DNA; given that the human genome has 3 billion base pairs, this implies that at most twelve million bytes of extra “software” transforms chimps into humans. And there is no suggestion in our evolutionary history that evolution found it more and more difficult to construct smarter and smarter brains; if anything, hominid evolution has appeared to speed up over time, with shorter intervals between larger developments. But leave aside for the moment the question of how to build smarter minds, and ask what “smarter-than-human” really means. And as the basic definition of the Singularity points out, this is exactly the point at which our ability to extrapolate breaks down. We don’t know because we’re not that smart. We’re trying to guess what it is to be a better-than-human guesser. Could a gathering of apes have predicted the rise of human intelligence, or understood it if it were explained? For that matter, could the 15th century have predicted the 20th century, let alone the 21st? Nothing has changed in the human brain since the 15th century; if the people of the 15th century could not predict five centuries ahead across constant minds, what makes us think we can outguess genuinely smarter-than-human intelligence? Because we have a past history of people making failed predictions one century ahead, we’ve learned, culturally, to distrust such predictions – we know that ordinary human progress, given a century in which to work, creates a gap which human predictions cannot cross. We haven’t learned this lesson with respect to genuine improvements in intelligence because the last genuine improvement to intelligence was a hundred thousand years ago. But the rise of modern humanity created a gap enormously larger than the gap between the 15th and 20th century. That improvement in intelligence created the entire milieu of human progress, including all the progress between the 15th and 20th century. It is a gap so large that on the other side we find, not failed predictions, but no predictions at all. Smarter-than-human intelligence, faster-than-human intelligence, and self-improving intelligence are all interrelated. If you’re smarter that makes it easier to figure out how to build fast brains or improve your own mind. In turn, being able to reshape your own mind isn’t just a way of starting up a slope of recursive self-improvement; having full access to your own source code is, in itself, a kind of smartness that humans don’t have. Self-improvement is far harder than optimizing code; nonetheless, a mind with the ability to rewrite its own source code can potentially make itself faster as well. And faster brains also relate to smarter minds; speeding up a whole mind doesn’t make it smarter, but adding more processing power to the cognitive processes underlying intelligence is a different matter. But despite the interrelation, the key moment is the rise of smarter-than-human intelligence, rather than recursively self-improving or faster-than-human intelligence, because it’s this that makes the future genuinely unlike the past. That doesn’t take minds a million times faster than human, or improvement after improvement piled up along a steep curve of recursive self-enhancement. One mind significantly beyond the humanly possible level would represent a Singularity. That we are not likely to be dealing with “only one” improvement does not make the impact of one improvement any less. Combine faster intelligence, smarter intelligence, and recursively self-improving intelligence, and the result is an event so huge that there are no metaphors left. There’s nothing remaining to compare it to. The Singularity is beyond huge, but it can begin with something small. If one smarter-than-human intelligence exists, that mind will find it easier to create still smarter minds. In this respect the dynamic of the Singularity resembles other cases where small causes can have large effects; toppling the first domino in a chain, starting an avalanche with a pebble, perturbing an upright object balanced on its tip. (Human technological civilization occupies a metastable state in which the Singularity is an attractor; once the system starts to flip over to the new state, the flip accelerates.) All it takes is one technology – Artificial Intelligence, brain-computer interfaces, or perhaps something unforeseen – that advances to the point of creating smarter-than-human minds. That one technological advance is the equivalent of the first self-replicating chemical that gave rise to life on Earth. If you travelled backward in time to witness a critical moment in the invention of science, or the creation of writing, or the evolution of Homo sapiens, or the beginning of life on Earth, no human judgement could possibly encompass all the future consequences of that event – and yet there would be the feeling of being present at the dawn of something worthwhile. The most critical moments of history are not the closed stories, like the start and finish of wars, or the rise and fall of governments. The story of intelligent life on Earth is made up of beginnings.

“Only a small community has concentrated on general intelligence. No one has tried to make a thinking machine and then teach it chess — or the very sophisticated oriental board game Go. [...] The bottom line is that we really haven’t progressed too far toward a truly intelligent machine. We have collections of dumb specialists in small domains; the true majesty of general intelligence still awaits our attack. [...] We have got to get back to the deepest questions of AI and general intelligence and quit wasting time on little projects that don’t contribute to the main goal.” - Marvin Minsky

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

Imagine traveling back in time to witness a critical moment in the dawn of human intelligence. Suppose that you find an alien bystander already on the scene, who asks: “Why are you so excited? What does it matter?” The question seems almost impossible to answer; it demands a thousand answers, or none. Someone who valued truth and knowledge might answer that this was a critical moment in the human quest to learn about the universe – in fact, the beginning of that quest. Someone who valued happiness might answer that the rise of human intelligence was a necessary precursor to vaccines, air conditioning, and the many other sources of happiness and solutions to unhappiness that have been produced by human intelligence over the ages. There are people who would answer that intelligence is meaningful in itself; that “It is better to be Socrates unsatisfied than a fool satisfied; better to be a man unsatisfied than a pig satisfied.” A musician who chose that career believing that music is an end in itself might answer that the rise of human intelligence mattered because it was necessary to the birth of Bach; a mathematician could single out Euclid; a physicist might cite Newton or Einstein. Someone with an appreciation of humanity, beyond the individual humans, might answer that this was a critical moment in the relation of life to the universe – the beginning of humanity’s growth, of our acquisition of strength and understanding, eventually spreading beyond Earth to the rest of the galaxy and the universe. The beginnings of human intelligence, or the invention of writing, probably went unappreciated by the individuals who were present at the time. But such developments do not always take their creators unaware. Francis Bacon, one of the critical figures in the invention of the scientific method, made astounding claims about the power and universality of his new mode of reasoning and its ability to improve the human condition – claims which, from the perspective of a 21st-century human, turned out to be exactly right. Not all good deeds are unintentional. It does occasionally happen that humanity’s victories are won not by accident but by people making the right choices for the right reasons. Why is the Singularity worth doing? The Singularity Institute for Artificial Intelligence can’t possibly speak for everyone who cares about the Singularity. We can’t even presume to speak for the volunteers and donors of the Singularity Institute. But it seems like a good guess that many supporters of the Singularity have in common a sense of being present at a critical moment in history; of having the chance to win a victory for humanity by making the right choices for the right reasons. Like a spectator at the dawn of human intelligence, trying to answer directly why superintelligence matters chokes on a dozen different simultaneous replies; what matters is the entire future growing out of that beginning. But it is still possible to be more specific about what kinds of problems we might expect to be solved. Some of the specific answers seem almost disrespectful to the potential bound up in superintelligence; human intelligence is more than an effective way for apes to obtain bananas. Nonetheless, modern-day agriculture is very effective at producing bananas, and if you had advanced nanotechnology at your disposal, energy and matter might be plentiful enough that you could produce a million tons of bananas on a whim. In a sense that’s what nanotechnology is – good-old-fashioned material technology pushed to the limit. This only begs the question of “So what?”, but the Singularity advances on this question as well; if people can become smarter, this moves humanity forward in ways that transcend the faster and easier production of more and more bananas. For one thing, we may become smart enough to answer the question “So what?” In one sense, asking what specific problems will be solved is like asking Benjamin Franklin in the 1700s to predict electronic circuitry, computers, Artificial Intelligence, and the Singularity on the basis of his experimentation with electricity. Setting an upper bound on the impact of superintelligence is impossible; any given upper bound could turn out to have a simple workaround that we are too young as a civilization, or insufficiently intelligent as a species, to see in advance. We can try to describe lower bounds; if we can see how to solve a problem using more or faster technological intelligence of the kind humans use, then at least that problem is probably solvable for genuinely smarter-than-human intelligence. The problem may not be solved using the particular method we were thinking of, or the problem may be solved as a special case of a more general challenge; but we can still point to the problem and say: “This is part of what’s at stake in the Singularity.” If humans ever discover a cure for cancer, that discovery will ultimately be traceable to the rise of human intelligence, so it is not absurd to ask whether a superintelligence could deliver a cancer cure in short order. If anything, creating superintelligence only for the sake of curing cancer would be swatting a fly with a sledgehammer. In that sense it is probably unreasonable to visualize a significantly smarter-than-human intelligence as wearing a white lab coat and working at an ordinary medical institute doing the same kind of research we do, only better, in order to solve cancer specifically as a problem. For example, cancer can be seen as a special case of the more general problem “The cells in the human body are not externally programmable.” This general problem is very hard from our viewpoint – it requires full-scale nanotechnology to solve the general case – but if the general problem can be solved it simultaneously solves cancer, spinal paralysis, regeneration of damaged organs, obesity, many aspects of aging, and so on. Or perhaps the real problem is that the human body is made out of cells or that the human mind is implemented atop a specific chunk of vulnerable brain – although calling these problems raises philosophical issues not discussed here. Singling out “cancer” as the problem is part of our culture’s particular outlook and technological level. But if cancer or any generalization of “cancer” is solved soon after the rise of smarter-than-human intelligence, then it makes sense to regard the quest for the Singularity as a continuation by other means of the quest to cure cancer. The same could be said of ending world hunger, curing Alzheimer’s disease, or placing on a voluntary basis many things which at least some people would regard as undesirable: illness, destructive aging, human stupidity, short lifespans. Maybe death itself will turn out to be curable, though that would depend on whether the laws of physics permit true immortality. At the very least, the citizens of a post-Singularity civilization should have an enormously

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

higher standard of living and enormously longer lifespans than we see today. What kind of problems can we reasonably expect to be solved as a side effect of the rise of superintelligence; how long will it take to solve the problems after the Singularity; and how much will it cost the beneficiaries? A conservative version of the Singularity would start with the rise of smarter-than-human intelligence in the form of enhanced humans with minds or brains that have been enhanced by purely biological means. This scenario is more “conservative” than a Singularity which takes place as a result of brain-computer interfaces or Artificial Intelligence, because all thinking is still taking place on neurons with a characteristic limiting speed of 200 operations per second; progress would still take place at a humanly comprehensible speed. In this case, the first benefits of the Singularity probably would resemble the benefits of ordinary human technological thinking, only more so. Any given scientific problem could benefit from having a few Einsteins or Edisons dumped into it, but it would still require time for research, manufacturing, commercialization and distribution. Human genius is not the only factor in human science, but it can and does speed things up where it is present. Even if intelligence enhancement were treated solely as a means to an end, for solving some very difficult scientific or technological problem, it would still be worthwhile for that reason alone. The solution might not be rapid, even after the problem of intelligence enhancement had been solved, but that assumes the conservative scenario, and the conservative scenario wouldn’t last long. Some of the areas most likely to receive early attention would be technologies involved in more advanced forms of superintelligence: broadband brain-computer interfaces or full-fledged Artificial Intelligence. The positive feedback dynamic of the Singularity – smarter minds creating still smarter minds – doesn’t need to wait for an AI that can rewrite its own source code; it would also apply to enhanced humans creating the next generation of Singularity technologies. The Singularity creates speed for two reasons: First, positive feedback – intelligence gaining the ability to improve intelligence directly. Second, the shift of thinking from human neurons to more readily expandable and enormously faster substrates. A braincomputer interface would probably offer a limited but real version of both capabilities; the external brainpower would be both fast and programmable, although still yoked to an ordinary human brain. A true Artificial Intelligence, or a human scanned completely into a sufficiently advanced computer, would have total self-access At this point one begins to deal with superintelligence as the successor to current scientific research, the global economy, and in fact the entire human condition; rather than a superintelligence plugging into the current system as an improved component. Often people instinctively and automatically adopt an “Us Vs. Them” view of this situation – the instinct that people who are different are therefore on a different side – but if humans and superintelligences are playing on the same team, it would be straightforward for the most advanced mind at any given time to offer a helping hand to anyone lagging behind. There is no technological reason why humans alive at the time of the Singularity could not participate in it directly. In our view this is the chief potential benefit of the Singularity to existing humans; not technologies handed down from above but a chance to become smarter and participate directly in creating the future. In history up until now, it has taken less and less time for major changes to occur. Life first arose around three and half billion years ago; it was only eight hundred and fifty million years ago that multi-celled life arose; only sixty-five million years since the dinosaurs died out; only five million years since the hominid family split off within the primate order; and less than a hundred thousand years since the rise of Homo sapiens sapiens in its modern form. Agriculture was invented ten thousand years ago; Socrates lived two and half thousand years ago; the printing press was invented five hundred years ago; the computer was invented around sixty years ago. You can’t set a speed limit on the future by looking at the pace of past changes, even if it sounds reasonable at the time; history shows that this method produces very poor predictions. From an evolutionary perspective it is absurd to expect major changes to happen in a handful of centuries, but today’s changes occur on a cultural timescale, which bypasses evolution’s speed limits. We should be wary of confident predictions that transhumanity will still be limited by the need to seek venture capital from humans or that Artificial Intelligences will be slowed to the rate of their human assistants (both of which I have heard firmly asserted on more than one occasion). We can’t see in advance the technological pathway the Singularity will follow, since if we were that smart ourselves we’d already have done it. But it’s possible to toss out broad scenarios, such as “A smarter-than-human AI absorbs all unused computing power on the then-existent Internet in a matter of hours; uses this computing power and smarter-than-human design ability to crack the protein folding problem for artificial proteins in a few more hours; emails separate rush orders to a dozen online peptide synthesis labs, and in two days receives via FedEx a set of proteins which, mixed together, self-assemble into an acoustically controlled nanodevice which can build more advanced nanotechnology.” This is not a smarter-than-human solution; it is a human imagining how to throw a magni-

“There’s this stupid myth out there that AI has failed, but AI is everywhere around you every second of the day. People just don’t notice it. You’ve got AI systems in cars, tuning the parameters of the fuel injection systems. When you land in an airplane, your gate gets chosen by an AI scheduling system. Every time you use a piece of Microsoft software, you’ve got an AI system trying to figure out what you’re doing, like writing a letter, and it does a pretty damned good job. Every time you see a movie with computer–generated characters, they’re all little AI characters behaving as a group. Every time you play a video game, you’re playing against an AI system.” - Rodney Brooks

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

fied, sped-up version of human design abilities at the problem. There are admittedly initial difficulties facing a superfast mind in a world of slow human technology. Even humans, though, could probably solve those difficulties, given hundreds of years to think about it. And we have no way of knowing that a smarter mind can’t find even better ways. If the Singularity involves not just a few smarter-than-usual researchers plugging into standard human organizations, but the transition of intelligent life on Earth to a smarter and rapidly improving civilization with an enormously higher standard of living, then it makes sense to regard the quest to create smarter minds as a means of directly solving such contemporary problems as cancer, AIDS, world hunger, poverty, et cetera. And not just the huge visible problems; the huge silent problems are also important. If modern-day society tends to drain the life force from its inhabitants, that’s a problem. Aging and slowly losing neurons and vitality is a problem. In some ways the basic nature of our current world just doesn’t seem very pleasant, due to cumulative minor annoyances almost as much as major disasters. This may usually be considered a philosophical problem, but becoming smarter is something that can actually address philosophical problems. The transformation of civilization into a genuinely nice place to live could occur, not in some unthinkably distant million-year future, but within our own lifetimes. The next leap forward for civilization will happen not because of the slow accumulation of ordinary human technological ingenuity over centuries, but because at some point in the next few decades we will gain the technology to build smarter minds that build still smarter minds. We can create that future and we can be part of it. If there’s a Singularity effort that has a strong vision of this future and supports projects that explicitly focus on transhuman technologies such as brain-computer interfaces and self-improving Artificial Intelligence, then humanity may succeed in making the transition to this future a few years earlier, saving millions of people who would have otherwise died. Around the world, the planetary death rate is around fifty-five million people per year (UN statistics) - 150,000 lives per day, 6,000 lives per hour. These deaths are not just premature but perhaps actually unnecessary. At the very least, the amount of lost lifespan is far more than modern statistics would suggest. There are also dangers for the human species if we can’t make the breakthrough to superintelligence reasonably soon. Albert Einstein once said: “The problems that exist in the world today cannot be solved by the level of thinking that created them.” We agree with the sentiment, although Einstein may not have had this particular solution in mind. In pointing out that dangers exist it is not our intent to predict a dystopian future; so far, the doomsayers have repeatedly been proven wrong. Humanity has faced the future squarely, rather than running in the other direction as the doomsayers wished, and has thereby succeeded in avoiding the oft-predicted disasters and continuing to higher standards of living. We avoided disaster by inventing technologies which enable us to cope with complex futures. Better, more sustainable farming technologies have enabled us to support the increased populations produced by modern medicine. The printing press, telegraph, telephone, and now the Internet enable humanity to apply its combined wisdom to problem-solving. If we’d been forced to move into the future without these technologies, disaster probably would have resulted. The technology humanity needs to cope with the coming decades may be the technology of smarter-than-human intelligence. If we have to face challenges like basement laboratories creating lethal viruses or nanotechnological arms races with just our human intelligence, we may be in trouble. Finally, there is the integrity of the Singularity itself to safeguard. This is not necessarily the most difficult part of the challenge, compared to the problem of creating smarter-than-human intelligence in the first place, but it needs to be considered. It is possible that the integrity of a human-originating Singularity needs no safeguarding; that any human from Gandhi to Stalin, if enhanced sufficiently far beyond human intelligence, would end up being wiser and more moral than anyone alive today. It’s also possible that a mistake in enhancement – hacking the brain incorrectly – could do terrible damage, catastrophic damage, or even irrecoverable damage. An analogous problem exists for Artificial Intelligence, where the task is not enforcing servitude on the AI or coming up with a perfect moral code to “hardwire”, but rather transferring over the features of human cognition that let us conceive of a morality improving over time (see the Singularity Institute’s section on Friendly Artificial Intelligence for more information, online at www.singinst.org). Safeguarding the integrity of the Singularity is another reason for facing the challenge of the Singularity squarely and deliberately. Safe human intelligence enhancement is an art that does not presently exist. Likewise the art of ethical Artificial Intelligence. In both cases, we can best safeguard the integrity of the Singularity by confronting the Singularity intentionally and with full awareness of the responsibilities involved. Despite the enormity of the Singularity, sparking the Singularity – creating the first smarter-than-human intelligence – is a problem of science and technology. The Singularity is something that we can actually go out and do – and do correctly, or alternatively screw up. It is not a philosophical way of describing something that inevitably happens to humanity. The sweep of human progress and our technological economy create the potential for the Singularity (just as it takes the entire framework of science to create the potential for a cancer cure), but it also takes a deliberate effort to run the last mile and fulfill that potential. If someone asks you if you’re interested in donating to AIDS research, you might reply that you believe that cancer research is relatively underfunded and that you are donating there instead; you would probably not say that by working as a stockbroker you support the world economy in general and thereby contribute as much to humanity’s progress toward an AIDS cure as anyone. In that sense, sparking the Singularity is no different from any other grand challenge – someone has to do it. The Singularity Institute’s mid-term mission is solving the problem of reflectivity – creating an AI that thinks about how to think. Just as Artificial Intelligence is probably the premier unsolved problem of modern science, reflectivity is the premier unsolved problem in Artificial Intelligence. Humans seem to get a tremendous amount of mileage out of thinking about how to think, but modern AI systems can’t seem to handle this at all. Even the theoretical foundations, the basic math of probability and decision theory, break down

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

when they try to describe an AI modifying the part of itself that modifies itself. A formal theory of reflectivity, we believe, is how you would go about building a rigorously safe, self-modifying Friendly Artificial Intelligence. This is the foundational work that can be done now, with limited funding; human intelligence enhancement would require far more funding, being on the scale of medical research rather than computer science research (and would also run into severe problems getting approval from ethics boards or the FDA). This is foundational work that must be done now; new basic math takes years or decades to develop. At this moment in time, there is a tiny handful of people who realize what’s going on and are trying to do something about it. It is not quite true that if you don’t do it, no one will, but the pool of other people who will do it if you don’t is smaller than you might think. If you’re fortunate enough to be one of the few people who currently know what the Singularity is and would like to see it happen – even if you learned about the Singularity just now – we need your help because there aren’t many people like you. This is the one place where your efforts can make the greatest possible difference – not just because of the tremendous stakes, though that would be far more than enough in itself, but because so few people are currently involved. The Singularity Institute is a 501(c)(3) tax-exempt nonprofit which exists to unite the efforts of the Singularity aware: to accelerate the arrival of the Singularity in order to hasten its human benefits; to close the window of vulnerability that exists while humanity cannot increase its intelligence along with its technology; to protect the integrity of the Singularity by ensuring that those projects which finally implement the Singularity are carried out in full awareness of the implications and without distraction from the responsibilities involved; and to code the damn AI, because someone has to do it. That’s our dream. Whether it actually happens depends on whether enough people take the Singularity seriously enough to do something about it – whether humanity can scrape up the tiny fraction of its resources needed to face the future deliberately and firmly. We can do better. The future doesn’t have to be the dystopia promised by doomsayers. The future doesn’t even have to be the flashy yet unimaginative chrome-and-computer world of traditional futurism. We can become smarter. We can step beyond the millennia-old messes created by human-level intelligence. Humanity can solve its problems – both the huge visible problems everyone talks about and the huge silent problems we’ve learned to take for granted. If the nature of the world we live in bothers you, there is something rational you can do about it. Don’t be a bystander at the Singularity. You can direct your effort at the point of greatest impact – the beginning. For more information about the Singularity, see the website of the Singularity Institute for Artificial Intelligence at: singinst.org Recommended for more information: singinst.org/AIRisk.pdf - More information on Artificial Intelligence, what needs to be done in this area, why the Singularity Institute focuses on AI. singinst.org/Biases.pdf - Thinking rationally about giant challenges such as the Singularity.

“One consideration that should be taken into account when deciding whether to promote the development of superintelligence is that if superintelligence is feasible, it will likely be developed sooner or later. Therefore, we will probably one day have to take the gamble of superintelligence no matter what. But once in existence, a superintelligence could help us reduce or eliminate other existential risks, such as the risk that advanced nanotechnology will be used by humans in warfare or terrorism, a serious threat to the long-term survival of intelligent life on earth. If we get to superintelligence first, we may avoid this risk from nanotechnology and many others. If, on the other hand, we get nanotechnology first, we will have to face both the risks from nanotechnology and, if these risks are survived, also the risks from superintelligence. The overall risk seems to be minimized by implementing superintelligence, with great care, as soon as possible.” - Nick Bostrom

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

SINGULARITY INSTITUTE FOR ARTIFICIAL INTELLIGENCE Tyler Emerson here, curator of the Singularity Summit and executive director of the Singularity Institute for Artificial Intelligence (SIAI). I want to tell you about SIAI: what we do, who we are, why we exist, what we want to do, and why you should care. Please set aside some time to read this, and let me know what you think, regardless of your views: [email protected].

What do we do? In the coming decades, humanity will likely create a powerful AI. We exist to confront this urgent challenge, the opportunity and risk. That may sound strange to some of you. AI researchers over the past 50 years have often over-promised and under-delivered. You should be skeptical, certainly, but hopefully not closed to new evidence. In our judgment, we see signs that AI systems of a new generation of power may come about in the next few decades – systems that understand their own behavior and work to improve themselves. This would create new technologies with tremendous value in education, medicine, science, engineering, and many other areas of human endeavor. However, there may also be substantial dangers – risks that would not be plainly apparent or simple to prevent. We have three principal aims: 1) Create the science to analyze the likely consequences of this nascent technology. 2) Create a clear vision of a desirable future. 3) Develop a roadmap for guiding the development of this technology to bring about that desirable future. And three major goals: 1) Foster scientific research on safe AI through research and development, fellowships, grants, and science education. 2) Further the understanding of its implications to society through educational outreach, such as our Singularity Summit. 3) Advance education among students to develop an interdisciplinary community of talented young scientists studying safe AI. Our present work: 1) In-house research (summarized later) 2) Research fellowships (e.g. Eliezer Yudkowsky) 3) Singularity Summit and monthly salon dinners 4) Educational outreach and awareness-building 5) OpenCog Initiative (early planning stage) 6) AI Impact Initiative (early planning stage) 7) Research grants (once seed funding secured)

What evidence is there for AI?



Some people say that computers can never show true intelligence, whatever that may be. But it seems to me that if very complicated chemical molecules can operate in humans to make them intelligent, then equally complicated electronic circuits can also make computers act in an intelligent way. And if they are intelligent, they can presumably design computers that have even greater complexity and intelligence. – Stephen Hawking, Lucasian Professor of Mathematics, University of Cambridge I asked Steve Omohundro, one of our Singularity Summit speakers, the question “What evidence is there for AI?” His reply: Unfortunately, “evidence” probably isn’t yet the right thing to be looking for when it comes to AI. Most AI researchers believe there are still some fundamental ideas missing in our understanding of intelligence. The only evidence that these will be discovered soon is the intuition of researchers. However, neuroscience is progressing steadily in its ability to model the biochemistry of the brain. It makes sense to ask for evidence that brain scanning technology will reach a certain point by a certain date. Even if new ideas in AI are not forthcoming, we are likely to be able to simulate brains in the next few decades and that is likely to give us tremendous insights into intelligence. If there is any possibility that AI might happen in our lifetimes, then it is important to understand the consequences, and to start planning for them now.

Who are we?



I have recently joined the SIAI Board of Directors, because the Singularity Institute is playing a critical role in advancing humanity’s understanding of the profound promise and peril of AI – an innovation that will significantly expand the power of intelligence. – Ray Kurzweil, Award-Winning Inventor & Author of The Singularity Is Near

We are a 501(c)(3) nonprofit institute in the Bay Area. We are still small. We have two researchers, Director of Research Dr. Ben Goertzel and Research Fellow Eliezer Yudkowsky. We have a small amount of funding to hire at least one more researcher for one year. I am

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

the only paid non-researcher; Director of Outreach Bruce Klein volunteers. Ray Kurzweil, author of The Singularity Is Near, is an SIAI Director. Our advisors include Peter Thiel, president of Clarium and director at Facebook; Dr. Barney Pell, CEO of Powerset; Dr. Nick Bostrom, director of the Oxford Future of Humanity Institute; Dr. Stephen Omohundro, president of Self-Aware Systems; and others. We have a few hundred small donors and a few dozen volunteers spread around the world at this time.

Why should you care? Please read “What is the Singularity?” included this handout, and then let me know what you think.

What do we want to do? A small selection, funding allowing: 1) Support the most promising scientists and build the premier program for safe AI research. 2) Develop a well-resourced, interdisciplinary scientific field and community for the study of safe AI. 3) Begin the SIAI Research Grant Program to support the most promising work on safe AI, existential risks, and reflectivity. 4) Craft a persuasive, accurate story, supported by a remarkable slideshow, to make a compelling case for our work. 5) Create and test educational outreach strategies at universities (starting at Stanford) to build awareness among gifted students. 6) Support the creation of a scientific journal and (bi)annual technical conference for general AI research and AI ethics. 7) Develop the OpenCog Initiative (an open-source framework for general AI research) and AI Impact Initiative (see later). 8) Build an initial SIAI Endowment that generates enough return to provide at least $1 million annually in working capital. 9) Build a research fellowship endowment to support-in-perpetuity six to 12 research fellows. If you want to learn more about our objectives, please contact me at [email protected] or 650-353-6063. Here’s more detail on some of the above items:

Research Program

Our mission is to create a framework for the development of safe AI. One of our paths toward this is research and development. We have three aims with our research program: 1) Understand the problems underlying the creation of safe AI with powerful general intelligence. 2) Pursue in-house theoretical and experimental research to work on the foundational problems of safe AI. 3) Provide the AI community at large with conceptual, mathematical, and software tools to move their work toward positive outcomes. Our aims are explicitly different than contemporary academic and industry AI research communities, in two key ways: 1) Our focus on general intelligence (e.g. reflective thought), rather than narrow AI software, such as chess-playing or fraud detection. 2) Our focus on beneficial use and safety, which must be foundational, not tacked on at the end of the theory and development process. See the first appendix to learn more about our research interests and our full program proposal, as well as the following from Eliezer.

The Problem of Reflectivity From Eliezer Yudkowsky: Our mid-term mission is solving the problem of reflectivity – creating an AI that thinks about how to think. Just as AI is probably the premier unsolved problem of modern science, reflectivity is the premier unsolved problem in AI. Humans seem to get a tremendous amount of mileage out of thinking about how to think, but modern AI systems can’t seem to handle this at all. Even the theoretical foundations, the basic math of probability and decision theory, break down when they try to describe an AI modifying the part of itself that modifies itself. A formal theory of reflectivity, we believe, is how you would go about building a rigorously safe, self-modifying safe AI. This is foundational work that can be done now, with limited funding; intelligence enhancement would require far more funding, being on the scale of medical research rather than computer science research (and would also run into severe problems getting approval from ethics boards or the FDA). This is the foundational work that must be done now; new basic math takes years or decades to develop.

Second Research Fellow We are looking for a research fellow to help hammer out the basic theory of self-improving, motivationally stable AI. This is an extremely difficult problem. You need to be patient enough to bang your head against it for as long as it takes. It will save time if you have already learned decision theory, probability theory, and mathematical logic. We will also look favorably upon anyone who has already learned a lot of neuroscience, evolutionary biology, physics, information theory, computer science, cognitive psychology, nonGOFAI, and anything else you can successfully argue as relevant. You must show evidence that you are technically competent before

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

we can consider whether you may be capable of original discoveries. You need to be a math talent who can think in code. You should have extraordinarily high fluid intelligence, since you will need to learn, unlearn, and do it fast. You should also be emotionally stable. All else is negotiable! For further details, please visit: http://www.singinst.org/aboutus/opportunities

Remarkable Slideshow



The ideas of the Singularity and Friendly AI are powerful, but more importantly, relevant today. I am a benefactor and advisor to the Singularity Institute for Artificial Intelligence because they are making unique contributions to these critical areas of knowledge. – Peter Thiel, Founder and President, Clarium Capital

Our ideas have yet to be presented in a way that resonates with a diverse group of people. We need to tell an accurate, compelling story that connects intellectually and emotionally with hundreds of thousands of people around the world. Have you seen Al Gore’s film documentary, An Inconvenient Truth? Whether you were persuaded by his case for climate change, it is reasonable to say that his arguments were strengthened by his extraordinary visuals – whether the beautiful graphics, humorous videos, or remarkable photos. Funding allowing, we would work with Duarte Design (in the Bay Area) on this project, or a similar group specializing in visual storytelling. We are especially interested in Duarte since they created Gore’s stunning slideshow visuals and helped him craft his message. Once created, the Singularity Institute would begin giving this slideshow around the country, focusing especially at top universities, since many more students need to be inspired to study these subjects, and ultimately devote their careers to them. This would help build a community of smart young scientists who have the multidisciplinary knowledge needed to work on the relevant problems; and help build our organizational infrastructure to support them. The compelling presentation and an accompanying website would also create a stronger case for the feasibility, desirability, and immediate relevance of our mission and goals, which is essential to increasing support.

AI Impact Initiative The AI Impact Initiative would foster a framework for the safe development of advanced AI. This technology has the potential to impact every aspect of human life. We are in a critical window of opportunity where we have powerful but temporary leverage to influence the outcome. Only a small group of scientists are aware of the core issues, and it is essential to get a broader range of thinkers involved. Funding allowing, we would organize an initial meeting to analyze the central issues, with a multidisciplinary group that brought a broad perspective, ranging from computer science, security, economics, evolutionary biology, cognitive science, political theory, decision theory, physics, philosophy, ethics, etc. Our near-term goal would be to create documents that clearly expressed the central issues, and disseminate them to students, scholars, and scientists. Our long-term goal would be to lay the foundation for a new multidisciplinary science to study these issues. This would involve creating expository materials, building an international network of scientists and scholars, organizing workshops, and creating a comprehensive report to provide direction for future research and development.

Research Grant Program SIAI and the Oxford Future of Humanity Institute have co-created a research grant program proposal, with funding being sought for the first two years. Research grants would be awarded to Ph.D. students or postdocs pursuing promising research in one of our focus areas: 1) Foundations of Reflection 2) Existential Risks 3) Friendly Artificial Intelligence This kind of program needs to exist. Each area is severely underfunded by governments, universities, and foundations. We would support novel research, not funded by traditional institutions, that advances scientific understanding in the focus areas. Grants would be awarded initially for three months, six months, or one year. Ph.D. student grants would be $5,000-$30,000; postdocs, $10,000-$80,000. We would have no preference to project location, institutional affiliation, country of residence, or nationality. Applications would be reviewed in two stages: 1) screened against our evaluation criteria, and then notified, three months after the deadline; and 2) sent to an external panel of experts for review if they passed the screening. Recommendations would then be presented to our final selection board with members from SIAI and Oxford Future of Humanity Institute, which would determine by vote which applications were rewarded. Applications would be evaluated on the quality of the proposed research and its potential contribution to knowledge in one of our areas. We would look for three characteristics: 1) A well-defined research problem, objective, or question 2) A sound assessment of what the research could achieve 3) A clear explanation of the significance of the research See the second appendix to learn more about each focus area.

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

What kind of help do we need?

1) Researchers – anyone who may have the talent to work on the technical challenges of safe AI. 2) A ton of volunteers to help grow this effort! It’s a fresh, rewarding, and intellectually-stimulating cause. 3) Help with marketing, PR, film and print media, donor development, outreach, and community-building. 4) Supporters who can organize dinner parties, cocktail parties, salons, discussion groups, or lectures. 5) An office space – bad. A place where we can hold working group and brainstorming sessions for 10-15 people. 6) Supporters to translate webpages into other languages, e.g. French, German, Japanese, Mandarin, Spanish, etc. 7) Contributors to the OpenCog Initiative or the AI Impact Initiative (both are in the early planning stage). 8) Contributors to the SIAI Slideshow Project, whether writing, editing, research, data, visuals, etc. 9) Contacts in academia, business, design, film industry, fundraising, marketing, nonprofit sector, etc. 10) Supporters to coordinate volunteers and donors, and engage in projects to increase our number of supporters. 11) Donations to the Singularity Institute – whether you can become a small or major donor, we need your support to grow. Please contact me if you can help: [email protected] or 650-353-6063.

Where can you learn more about our work? www.singinst.org/AIRisk.pdf – Artificial Intelligence as a Positive and Negative Factor in Global Risk www.singinst.org/Biases.pdf – Cognitive Biases Potentially Affecting Judgment of Global Risks www.singinst.org/media/ – Video interviews with staff and advisors + videos from the Singularity Summit at Stanford www.singinst.org/reading/ – Reference material on AI, cognitive science, mathematical and philosophical foundations, etc. www.singinst.org/donate/ – Where you can make a one-time or monthly donation, and learn about the SIAI Donor Network

What did we do in 2006?



SIAI is the only organization that exists for the expressed purpose of achieving the potential of smarter-than-human intelligence safer and sooner. It’s the organization best positioned to attract the top AI researchers and to act in the best interests of humanity based on a deep understanding of the power and behavior of intelligence, with freedom from any other influences. SIAI has already done more than any other organization towards investigating deeply into the problem and the potential impacts of a smarter-than-human intelligence. I believe that by acting now, you can give the institute the best chance of becoming the center of gravity for focusing effort on this crucial step of bringing a new form of intelligence into our world and that this is the best opportunity for making our world a genuinely nice place to live. – Edwin Evans, SIAI Major Donor

We co-sponsored the Singularity Summit at Stanford, conceived and organized by myself in collaboration with Ray Kurzweil, Professor Todd Davies of the Symbolic Systems Program at Stanford, and Stanford undergraduates Yonah Berwaldt and Michael Jin. In late 2006, we created the initial structure for the SIAI Research Grant Program (seed funding still needs to be secured), a program in partnership with the Oxford Future of Humanity Institute. Research Fellow Eliezer Yudkowsky released “Artificial Intelligence as a Positive and Negative Factor in Global Risk” and “Cognitive Biases Potentially Affecting Judgment of Global Risks,” publications forthcoming from Oxford University Press in the edited volume Global Catastrophic Risks (2008). Yudkowsky presented at the Bay Area Future Salon, Singularity Summit at Stanford, and AGI (Artificial General Intelligence) Workshop. We raised $200,000 through a challenge grant backed by Clarium Capital President Peter Thiel, following which Marcello Herreshoff began as a research associate, and Allison Taguchi joined as director of development. Neil Jacobstein, Dr. Stephen Omohundro, Dr. Barney Pell, and Peter Thiel joined the SIAI Board of Advisors. Marcello Herreshoff and Eliezer Yudkowsky worked together full-time on AI theory. Conversely, Yudkowsky had to spend less time presenting and writing publications. In July and August, Nick Hay (a Masters candidate at the University of Auckland) and Peter de Blanc (a math graduate from the University of Pennsylvania) worked with Herreshoff and Yudkowsky on AI for six weeks under an internship. Our 2006 research consisted of analyzing, in excruciating detail, many small “toy” problems that humans seem to solve using thinking about thinking, looking for a key insight that explains why these problems are tractable. These toy problems could be easily solved by writing a specialized AI program; but we are looking for the metaprogram that lets humans look over the problem and then write a program to solve it. Since metaprogramming is, in general, a difficult and unsolved problem, we are analyzing it for simple cases to make the problem tractable. The research of Yudkowsky and Herreshoff produced some interesting insights. We are looking to hire a senior science writer who, as part of their role at SIAI, would begin to report a backlog of past results that includes mathematical

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

definitions of optimization, criteria of intelligence within a Bayesian framework, and hidden downsides of randomized algorithms in Artificial Intelligence. APPENDIX I: RESEARCH PROGRAM PROPOSAL - RESEARCH AREAS By SIAI Director of Research Dr. Ben Goertzel



Here is a summary of our research program proposal. If you are interested in potentially supporting this work, please contact us. We have divided our proposal into two phases: Phase I: Engage in theoretical analysis and tool building in the attempt to create a viable framework for the development of Friendly AI. Phase II: Design and engineer Artificial General Intelligence (AGI) utilizing the theory, tools, and technologies created in Phase I. In Phase I, our mathematical and conceptual work would focus on: 1) Probability theory, decision theory, and algorithmic information theory to study AGI ethics 2) Creating a shared ontology for discussing concepts important for describing AGI designs and behaviors



In Phase I, our software tools and technology work would focus on: 1) A simulation world for AGIs 2) A collection of IQ and ethical test scenarios for AGI systems, implemented in the simulation world 3) A language for communicating with AGIs with minimal ambiguity 4) Knowledge resources allowing AGIs to more easily represent and reason about their own structure and dynamics 5) Key cognitive technologies designed to serve as flexible components within safe, beneficial AGI architectures



We realize that our stated research areas below are numerous; it is possible that SIAI may not achieve the funding or staffing required to address them all during the Phase I period. As this is R&D, it is not possible to accurately predict the amount of time or effort that would be required to complete a particular task to a particular level. What is given here is a moderately comprehensive list of R&D sub-projects lying within our general scope (previously outlined). Which topics are pursued in what order would depend on a number of practical factors including the rate of scaling of our R&D funding, and the interests and expertise of the particular research staff that we hire. The Singularity Institute’s research team would be a small one for the foreseeable future. We would focus initially on recruiting a small number of highly gifted individuals, each of whom would be capable of spanning multiple research areas. Fortunately, we have already identified a number of individuals who have expressed interest in joining our research team once sufficient funding is available.

Theoretical Research Research Area 1: Mathematical Theory of General Intelligence Our research in this area would focus on using algorithmic information theory and probability theory to formalize the notion of general intelligence. Important work in this area has been done by Marcus Hutter (a pioneer in universal artificial intelligence theory), Jürgen Schmidhuber (codirector of the Dalle Molle Institute for Artificial Intelligence in Switzerland), Shane Legg (Ph.D. student at the Dalle Molle Institute for Artificial Intelligence), and others, as well as by our team; but this work has not yet been connected with pragmatic AGI designs. Meeting this challenge would be one of our major goals going forward. Specific focus areas within this domain include: 1) Mathematical Formalization of the “Friendly AI” Concept. Proving theorems about the ethics of AI systems relies on possessing an appropriate formalization of the notion of ethical behavior on the part of an AI. This formalization is a difficult research question unto itself. 2) Implications of Algorithmic Information Theory for the Predictability of Arbitrarily Intelligent AIs. In 2006, Shane Legg made an interesting, but ultimately failed attempt to prove algorithmic information theoretic limitations on the possibility of guaranteeing ethical behavior on the part of future AIs. This line of research however has significant potential for future exploration. 3) Formalizing the Concept of General Intelligence. Shane Legg and Marcus Hutter published a paper in 2006 on a formal definition of general intelligence. Their work is excellent but can be extended; e.g., to connect their ideas with practical intelligence tests for AGIs. 4) Reflective Decision Theory: Extending Statistical Decision Theory to Strongly Self-Modifying Systems. Statistical decision theory, as it stands, tells us little about software systems that regularly make decisions to modify their own source code. This deficit must be remedied if we wish to formally understand self-modifying AIs, their potential dangers, and how to ensure their long-term safety and positive use.

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

5) Dynamics of Goal Structures Under Self-Modification. Under what conditions would an AGI system’s internal goal structure remain invariant as the system self-modifies? Suppose one of the system’s top-level goals would be goal-system invariance – it would still not guarantee invariance. Further conditions are needed, but the nature of these conditions has not been seriously investigated. This is a deep mathematical issue in the dynamics of computational intelligence, with critical implications for the development of safe AGIs.

Research Area 2: Interdisciplinary Theory of AGI An objective in this area would be to create a systematic framework for the description and comparison of AGI designs, concepts, and theories. We would also make contributions to the practicalities of creating, engineering, and understanding real-world AGI systems. 1) Mind Ontology: A Standardized Language for Describing AGI Systems and Related Concepts. One of the issues holding back AGI progress is that different researchers often use different languages to discuss the same things. One solution is to agree upon a standard ontology of AGI-related concepts. An initial draft of such an ontology exists, but needs extension and refinement. A description of current AGI designs and cognitive neuroscience knowledge in terms of the common ontology also needs to be undertaken. 2) AGI Developmental Psychology. Once an AGI is created, it will have to be taught: the first AGI will likely be more like an artificial baby than an artificial adult. Current theories of developmental psychology are focused on human psychological development. However, if AGI development begins by creating “baby AGI’s” and teaching them gradually, we will need a theory of AGI developmental psychology to guide our work. Recent theoretical work by Ben Goertzel and Stephan Vladimir Bugaj connected Piagetan developmental psychology with the theory of uncertain inference; but considerably more research is required. One of the issues here is the interdependence of ethical development with cognitive development, which is only moderately understood in humans, and will likely be quite different in AGIs.

Research Area 3: AGI Ethical Issues One of our core views is that ethical issues must be central to AGI research and development, rather than tacked on peripherally to AGI designs created without attention to ethical considerations. Several of our focus areas have direct implications for AGI ethics (particularly the investigation of goal system stability), but we would also heavily investigate several other issues related to AGI ethics, including: 1) Formalizing the Theory of Coherent Extrapolated Volition. Research Fellow Eliezer Yudkowsky has proposed “coherent extrapolated volition” (CEV) as a way of arriving at a top-level supergoal for an AI that represents the collective desires of a population of individuals. While fascinating, the idea has only been presented informally, and a mathematical formalization is necessary so that its viability could be assessed. For example, it is of interest to try to articulate formally the conditions under which the CEV of a population of individual agents, appropriately defined, would exist. This may depend on the coherence versus divergence of the beliefs or mind-states of the individuals. 2) Framework for Formalizing Desired Outcomes. To create safe and beneficial AI systems, we must have a clear vision of a desirable outcome. The recently developed science of Positive Psychology is making great strides in understanding elements that promote human happiness. Political philosophy has studied a variety of approaches to structure “the good society” in a way that maximizes the benefits to its citizens. We would work toward a framework that formalized these kinds of insights so that they could be considered for AI systems. 3) Decision-Theoretic and Game-Theoretic Foundations for the Ethical Behavior of AGIs. Microeconomics and decision theory study the nature of individual preferences and their influence on behavioral outcomes. Game theory is the core mathematical theory of decision making by interacting agents. We would use these tools to analyze the likely behavior of alternative models for the safe deployment of self-modifying AGIs. The preferences of an agent together with the behavior of other agents in its environment determine the actions it will take. We must design the preferences of agents so that their collective behavior produces the results we desire and is stable against internal corruption or external incursion.

Tools and Technologies Research Area 4: Customizing Existing Open-Source Projects Our work in this area would focus on customizing and developing existing open-source software projects. There are valuable, preexisting projects moving slowly due to lack of funding, which could be morphed into specific tools for aiding AGI development. Three examples are the AGISim simulation world project, the Lojban language for human-machine communication, and the Mizar mathematics database. Like any complex engineering challenge, building an AGI involves a large number of tools, some of which are quite complex and

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

specialized. One delay of progress in AGI is the lack of appropriate tools. Each team must develop their own, which is time-consuming and distracts attention from the actual creation of AGI designs and systems. One of the key roles the Singularity Institute could play going forward would be in the creation of robust tools for AGI development, to be utilized in-house and by the AGI research community at large. 1) AGISim, a 3D Simulation World for Interacting with AGI Systems. AGISim is an open-source project in alpha release. It is usable, but still needs more coding work done. A related task, of significant use to robotics researchers, would be the precise simulation of existing physical robots within AGISim. AGISim also could play a key role in some of the AGI IQ/ethics evaluation tasks to be described later. 2) Lojban: A Language for Communicating with Early-Stage AGIs. Lojban is a constructed language with hundreds of speakers, based on predicate logic. Thus, it is particularly suitable for communication between humans and AGIs. A Lojban parser exists, but needs to be modified to make it output logic expressions, which would then allow Lojban to be used to converse with logicbased AGI systems. This would allow communication with a variety of AI systems in a human-usable yet relatively unambiguous way, which would be valuable for instructing AGI systems, including ethical behavior instruction. 3) Translating Mizar to KIF. Mizar is a repository of mathematical knowledge, available online but in a complex format that is difficult to feed into AI theorem-proving systems. In six months, a qualified individual could translate Mizar to KIF, a standard predicate logic format, which would enable its use within theorem-proving AI systems, a crucial step toward AGI systems that can understand themselves and the algorithms utilized within their sourcecode. Research Area 5: Design and Creation of Safe Software Infrastructure Some key areas of tool development are not adequately addressed by any current open-source project, for example, the creation of programming languages and operating systems possessing safety as built-in properties. Of course, Singularity Institute researchers would not be able to complete such large, complex projects on their own, but the institute could potentially play a leadership role by articulating detailed designs, solving key conceptual problems, and recruiting external partners to assist with engineering and testing. 1) Programming Languages Combining Efficiency with Provability of Program Correctness. In the interest of AGI safety, it would be desirable if AGI software programs could be proved to correctly implement the software designs they represented. However, there is no language that supports proof-based program correctness checking and is sufficiently efficient in terms of execution to be pragmatically useful for AGI purposes. Such a programming language framework would require major advances in programming language theory. 2) Safe Computer Operating Systems. Is it feasible to design a provably correct OS? In principle, yes, but it would require a programming language that combined efficiency with provable correctness, as well as several interconnected breakthroughs in OS theory. Creating a version of Unix in a programming language that supported provable correctness would be a start, but there would be many issues to address. This research would require close collaboration between a mathematician and an experienced operating systems programmer.

Research Area 6: AGI Evaluation Mechanisms Safe AGI development would be hastened if there were well-defined, widely-accepted means of assessing general intelligence, safety, and beneficial use. The provision of such means of assessment is a tractable task that fits squarely within the core mission of SIAI. A few comments regarding AGI intelligence testing must be inserted here, as general context. IQ tests are a controversial but somewhat effective mechanism for assessing human intelligence. Narrow AI software is evaluated by a variety of mechanisms appropriate to the various domains in which it operates. AGI software, on the other hand, is not currently associated with any generally accepted evaluation mechanism. The Turing Test and variations purport to assess the effectiveness of AGI systems at emulating human intelligence, but have numerous shortcomings: not all AGI systems necessarily aim at the emulation of human intelligence; and furthermore, these tests do not provide any effective way of assessing the continual progress of AGIs toward more and more general intelligence. The Loebner Prize, a chat-bot contest, purports to assess the progress of AI systems toward general intelligence in a conversational fluency context, but its shortcomings have been well documented. It is with this background in mind that we propose to devote some effort to the creation of intelligence evaluation mechanisms focused specifically on AGI. We do not expect this to lead to any single, definitive “AGI IQ test,” but rather to a host of evaluation mechanisms that are useful to AGI researchers in assessing and comparing their systems. Among the most innovative and powerful mechanisms we suggest are ones involving assessing the behavior of AGIs within the AGISim simulation world. Assessing the ethicalness of an AGI’s behavior and cognition is a matter even less studied. Our primary focus in this regard would be on the creation of “ethical behavior rubric” in the form of scenarios within the AGISim world. This sort of assessment does not provide any sort of absolute guarantee of an AGI system’s safety or beneficial application, but nevertheless would allow a far more

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

rigorous assessment than any approach now available. We consider it important that work in this area start soon, so that “ethics testing” becomes accepted as a standard part of AGI R&D. 1) Recognizing Situational Entailment Challenge. We want to extend the “Recognizing Textual Entailment” challenge by defining a “Recognizing Situational Entailment” challenge, in which AI systems are challenged to answer simple English questions about “simulation world movies” that they are shown. The movies would be generated using the AGISim framework. An annual workshop to address this challenge could be organized as part of a recognized AI conference. 2) Development of a Suite of Benchmark Learning Tasks within AGISim. Within the context of the AGISim world, we would develop a set of tasks on which any AGI system could be tested, e.g. playing tag, imitating behaviors, imitating structures built from blocks, etc. Having a consistent set of benchmark tasks for comparing different AGI approaches is important for coordination of progress in the field. 3) Development of a Suite of Benchmark Ethics Tests within AGISim. Just as one could test intelligence through AGISim scenarios, one could also test ethics, by placing the AGI in situations where it must interact with other agents, assessing the ethical sensitivity of its behaviors. Testing within such scenarios should become a standard part of assessing the nature of any new AGI architecture. 4) Porting of Human IQ Tests to AGIs. To what extent are human IQ tests overly human-centric? Could we create variants of the IQ tests administered to humans that are more appropriate for AIs? It may be that different variants must be created for different AIs, e.g. based on the nature of the AIs embodiment and sensory organs. Investigating the variation of IQ questions, based on the nature of the intelligent system being tested, would be one way to probe the core of intelligence.

System Design and Implementation Research Area 7: AGI Design This is arguably the most critical component of the path to AGI. AGI design and engineering would be our central focus in Phase II. In Phase I, however, our work in this area would focus on the comparison and formalization of existing AGI designs. This is crucial, as it would lead to a better understanding of the strong and weak points in our present understanding of AGI (and our understanding of Friendly AI), and form the foundation for creating new AGI designs, as well as analyzing and modifying existing AGI designs. 1) Systematic Comparison of AGI Designs. A number of designs for AGI have been proposed, some in the public literature, with varying levels of detail. What are their major overlaps, major common strengths, and major common weaknesses? The first step toward resolving this may be to describe the various systems using a common vocabulary, such as the Mind Ontology Project.

Research Area 8: Cognitive Technologies Our in-house research and development plans are founded, in part, on the premise that the appropriate use of probability theory is likely to play an important role in the development of safe, beneficial AGI. With this in mind, the “cognitive technologies” aspect of our Phase I centers on the creation of several cognitive components utilizing probability theory to carry out operations important to any AGI. Our research would differ from most work on probabilistic AI due to our focus on generality of scope rather than highly specialized problem-solving. In order to reason probabilistically about real-world situations, including situations where ethical decisions must be made, powerful probabilistic reasoning tools would be needed, and tools different-in-kind than ones currently popular for narrow-AI applications. 1) Efficient Techniques for Managing Uncertainty in Large Dynamic Knowledge Bases. Using Bayesian probability techniques is not sufficiently computationally efficient to be a pragmatic approach to AGI on present systems. Approximations are needed, which achieve efficiency without losing too much accuracy. A variety of approaches are possible here, and need to be fleshed out mathematically and computationally, and compared to each other. For example, work on loopy Bayes nets, imprecise and indefinite probabilities, and probabilistic logic networks is relevant here. 2) Probabilistic Evolutionary Program Learning. One of the more powerful optimization techniques available is “probabilistic evolutionary learning,” or Estimation of Distribution Algorithms (EDAs). Recent research by Dr. Moshe Looks (software engineer at Novamente LLC) has extended EDAs to automated program learning, but the state of the art only allows automated learning of relatively simple programs. Extension of this paradigm is necessary, to allow learning of programs involving recursion and other complex programming constructs. 3) Probabilistic Inference Driven Self-Modification of Program Code. Is it possible to write code that uses probabilistic reasoning to model its own behavior, and then modifies itself accordingly? Two proprietary AGI designs, Novamente LLC and Self-Aware Systems, use aspects of this idea. However, there is no general theory covering this kind of algorithm, and many possible approaches may be viable.

PA L AC E O F FINE A RTS T H E AT RE • SAN F R ANCI SCO, CA • SE PT E MB E R 8- 9, 20 07

APPENDIX II: RESEARCH GRANT PROGRAM PROPOSAL - FOCUS AREAS By SIAI Research Fellow Eliezer Yudkowsky and SIAI Advisor Dr. Nick Bostrom

Foundations of Reflection Focus Area In 1965, the statistician I. J. Good suggested that any sufficiently smart mind could design a next generation of smarter minds, giving rise to a positive feedback loop and an “intelligence explosion.” In purest form, an AI could rewrite its source code, or design new hardware for itself. Taking this scenario seriously presents us with foundational mathematical challenges: modern logics, probability theories, and decision theories do not adequately handle this type of self-reference. These difficulties are symptomatic of a more fundamental problem – current AI techniques make little use of reflection. We, however, derive a great deal of benefit from thinking about thinking. What makes self-reference tractable for us? Can the same techniques be applied to AI and made reliable, and safe? The Foundations of Reflection Focus Area would support research on the following: 1) Decision-theoretic, probability-theoretic, or logical foundations for self-referential agents. 2) Analyses of the stability of optimization targets, utility functions, and choice criteria in self-modifying agents. 3) Techniques for self-modification and self-improvement in AI that promise to be strengthenable to extreme reliability. 4) Formal attempts to prove the problem of ensuring extreme reliability in AI unsolvable, or unsolvable using a particular approach, using proof-theoretic or other mathematical arguments. 5) Experiments to determine how human problem-solvers use reflection. 6) Other work that demonstrates a significant new idea or approach.

Existential Risks Focus Area Nearly 99.9% of all species that ever lived are now extinct. Will our own species have the same end? How could that happen? And what can we do to stave off the end? An “existential risk” is defined as one that threatens to annihilate Earth-originating intelligent life or permanently and drastically curtail its potential. Existential risks are the extreme end of global catastrophic risks. The exact probability of an existential disaster in this century is unknown, but there is reason to think that it is significant. That, at least, is the consensus among the small number of serious scholars who have investigated the question to date: 50%, Professor Sir Martin Rees, President of the Royal Society (Our Final Hour, 2003); 30%, Professor John Leslie (End of the World, 1996); significant, Judge Richard Posner (Catastrophe, 2004); and not less than 25%, Dr. Nick Bostrom (“Existential Risks: Analyzing Human Extinction Scenarios,” 2002). The Existential Risks Focus Area would support research on the following: 1) Original studies and metaresearch on existential risks. 2) Comparative analyses of the threats posed by existential risks. 3) Methodological improvements for studying existential risks. 4) Analyses of complex ethical issues related to existential risks (such as the weight to be placed on the interests of future generations, and how to allocate moral responsibility for risk-reduction). 5) Identification of common elements that contribute to existential risks.

Friendly AI Focus Area The Foundations of Reflection Focus Area arises as part of our broader research interest in Friendly AI: handling the challenge and risk of smarter-than-human AI. We would support the most promising technical research on this subject: for example, identifying the problems of learning a multicomponent utility function, under conditions of uncertainty, from noisy, non-independent observations. The Friendly AI Focus Area would support research on the following: 1) Conditions sufficient, or necessary, for an agent to learn information in a complexly structured utility function or other choice criterion. 2) Acceptable ways of transforming human motivational structure (e.g. adaptation aspiration) into normative criteria, e.g. utility functions. 3) Detailed identification of known biases or logical fallacies in previously published work on AI motivations. 4) Rigorous analytic philosophy that addresses some of the biggest challenges in Friendly AI, e.g. “How do we know what we want?” 5) Clear explanations of how an existing AI methodology would fail in Friendliness when scaled to superintelligence, selfmodification, real-world capabilities exceeding the programmers, metamoral questions, or other long-term challenges of Friendly AI.