Teaching Helicopters To Fly

Credit: sxc.hu

by TAYLOR CONE

ENG + TECH

Teaching Helicopters to Fly

A radio-controlled helicopter, glistening bright red in a baby blue sky, carries out stunning maneuvers – rolls, flips, inverted hovers, and more. The air showcaliber routine continues for several minutes, each maneuver executed perfectly and without mistakes. After the show, you walk over to the pilot to congratulate him or her on an outstanding performance only to find that a computer was in complete control for the duration of the show. Sound like a dream? Thanks to recent developments in machine learning and artificial intelligence this scenario is no longer science fiction. Stanford researchers have developed new algorithms that allow computers to “learn” Through complex logic and countless lines of code, some computer systems have the ability to learn from the patterns they observe. Under the direction of Computer Science Professor Andrew Ng, Ph.D. student Adam Coates has spent the last five years developing an “apprenticeship learning” system that involves a human “teacher” from which the computer observes and learns. Coates, along with fellow Ph.D. students Pieter Abbeel, Timothy Hunter, and Morgan Quigley, use radio-controlled (RC) helicopters as the test bed for the project. In

particular, they study the controls necessary to maintain stable flight as well as perform stunning maneuvers. With several onboard sensors, two ground-based cameras and a few computers doing the brute force data analysis, the control system has learned to execute some impressive aerobatics.

Can Computers Learn

Researchers develop learning algorithms for helicopter aerobatics

The advent of artificial intelligence brought a number of revolutionary concepts to our understanding of what computers can do. One such concept is machine learning, which involves the design and development of algorithms that allow computers to learn from the patterns found in recollected data. This grants computers abilities never before utilized and has implications on the areas of robotics, manufacturing, warfare and search and rescue among others. Conventionally, programmers write codes that explicitly outline every command to a computer. This is useful and efficient in most contexts and has had great success throughout the history of computing. However, with more complex control problems, such explicit coding is extremely inefficient. In these situations, it may make more sense to turn to machine learning techniques, where the computer has the ability to determine the commands by its own.

volume VIII 29

Credit: Andrew Ng

Snapshots of the flight of the helicopter during a complicated maneuver.

For example, imagine a scenario as simple as picking up a pen. Think of all the commands your brain sends your muscles during this action. Your brain considers how far away the pen is, when to open your fingers, which fingers to use, how tightly to grasp it, which muscles to employ, how much to contract these muscles, the speed at which to execute all of this, and much more. If we were conscious of every command going through our brain at a given moment, we would not be able to focus on anything else. Similarly, if we are able to minimize the amount of explicit coding we put into a program, it will be that much easier to understand and control. This is exactly what Coates and his colleagues have in mind: if they can simply write robust learning algorithms, their programs will be able to figure out what to do in specific situations on their own.

Apprenticeship Learning

Ng’s research group chose to pioneer a new type of machine learning for their autonomous helicopter project. Rather than using conventional machine learning that involves the computer running a simulation and learning from the resulting patterns, they integrated a human element.

The breadth of the applications of these learning algorithms is only limited by our willingness to teach our skills to computers. 30 www.stanfordscientific.org

Expert RC helicopter pilot Garett Oku acts as the team’s teacher – not teaching the students how to fly, but teaching the helicopters. Having a human teacher, particularly such an experienced pilot like Oku, “made us realize that when you have this slight advantage of having someone who can help you with the learning process, that is a huge boon,” Coates says. Since humans can already fly RC helicopters with relative ease, it seems logical to replace computer-run simulations with human pilot mentors. As Coates describes, “we set out to use learning algorithms to try to learn to

fly the helicopter the way a human could to demonstrate that rather than solving this using classical control methods you could use artificial intelligence algorithms to learn to fly this the way a human does.” “As soon as we had a human to teach the computer so the computer knew where to start, that completely changed the ball game,” Coates explains. Not only do they now have a human-controlled system for the computer to observe, but the responses – the inputs and outputs – are all from the real world rather than a simulation. Combined, these benefits make for a much more effective machine learning system.

Why Helicopter Aerobatics?

Although autonomous acrobatic helicopters are the most tangible result of this project, it is important to note that in the eyes of the researchers, helicopters are merely test beds for the development of their machine learning algorithms. So if they are just a test bed, what motivated the research team to use helicopters? Mostly, helicopters present a unique design challenge for they are inherently difficulty to control. “If you’re flying in an airplane and you let go of the controls...the airplane will tend to keep itself upright without any human interaction,” says Coates. The same is certainly not true for helicopters, which tend to flip over and crash without any human input. According to Coates, “they’re much more complicated and the behavior of a helicopter is just really difficult to predict.” If stable helicopter flight is already difficult enough, why make it even more complicated by pursuing aerobatics? “The reason we wanted to do aerobatics is because we can demonstrate these learning algorithms are able to solve really hard

control problems that until now have been completely inaccessible,” Coates explains. “Now we can actually meet the level of a human pilot, which is brand new.” This is the most intriguing part of the project: humans have already solved this extremely difficult control problem. Knowing little or nothing about aerodynamics, a human pilot can fly aerobatics with a radio-controlled helicopter. Humans “clearly have this system that’s able to fly helicopters well beyond anything that the best computers were able to do, yet as a control problem it’s still largely unsolved,” Coates adds.

The Hardware

Even though the project’s focus is software development, it is nonetheless imperative to have a robust physical platform with which to test the algorithms. In fact, the physical implementation can sometimes cause the most problems, as Coates describes: “We were worried that even if the learning software were good enough to do [what we wanted], we were concerned that sensor technology would just not be capable.” Fortunately, the hardware proved to be very successful. As for the airframe itself, it is simply an off-the-shelf helicopter, as are the engines and the rest of the propulsion system. The rest of the hardware – the gyroscopes, accelerometers and compasses – are all custom added and stored in a 100-gram black box on the helicopter. Inside the black box are three gyroscopes which sense angular rotation about three axes, three accelerometers which sense acceleration along the three linear axes, and a three-dimensional magnetic compass which senses the earth’s magnetic field. Together, these sensors give a relatively good idea of the orientation of the aircraft with respect to the earth.

To add position information, the team has a pair of cameras on the ground as well. These two cameras effectively act as eyes, tracking the helicopter in the sky and generating the data necessary for specific position knowledge. “All of these sensors get a coherent estimate in the computer of where the helicopter is and which way it’s pointing,” Coates explains.

The Software

Considering how many sensors are providing the computers with constant input, it is a veritable challenge interpreting it all and producing meaningful outputs. What does the helicopter’s “learning process” actually consist of? The first step of the process involves a human pilot – Oku in this case – conducting several aerobatic flights with the helicopter while the computer “observes.” During this time, the computer examines the pilot’s every input and the exact result. These results vary by trial due to weather conditions as well as changes in the system, and the computer records these minor differences among the trials. After enough trials, the computer can then differentiate the pilot’s intended trajectories and the actual resultant trajectories, thereby understanding the necessary commands for intended results in a wide variety of scenarios. “The basic approach is to have a human fly the helicopter while the computer observes how the helicopter behaves,” Coates says. “From that, the computer tries to learn what the human was trying to accomplish, the maneuvers they were trying to fly. Then, once we put the computer in control of the helicopter, it has to figure out what actions to execute in order to repeat what the human did.” How well the computer learns and accurately replicates the maneuvers carried

volume VIII 31

Credit: Andrew Ng

out by the human pilot is actually quite astounding. “We can fly air shows that are as good as those flown by our expert pilot,” Coates claims. Pretty impressive, considering the inherent complexity of the system being used. “These maneuvers represent by far the most advanced, most difficult maneuvers flown on any computer controlled helicopter,” Ng adds.

Firefighting and Autonomous Cartography

If the researchers pursued the possibility of extending the same principles into full-scale helicopters, these algorithms would have great implications in tasks such as firefighting. In fact, a more complex algorithm would be necessary for such a scenario, which would require consideration of the air currents associated with fire. “The interesting thing about the firefighting application is you have a lot of nasty air currents moving around that can affect the helicopter,” Coates explains. “Excellent pilots can compensate for this but it would be a little scary to have a hand-tuned control system flying because you have no idea what will happen when it gets hit with turbulence.”

“These maneuvers represent by far the most advanced, most difficult maneuvers flown on any computer controlled helicopter.” - Andrew Ng 32 www.stanfordscientific.org

If the team’s algorithm were able to watch an expert human pilot fly in such conditions, it would learn how to react to unforeseen circumstances and hopefully be able to do the same when placed in that environment. “Lately we’ve seen a lot of firefighting in California with helicopters and we’ve seen how dangerous that can be, and it’d be nice to get those people out of harm’s way,” Coates says. Another useful application is in autonomous flight and aerial mapping and cooperation between air and ground vehicles. Imagine there is a ground vehicle that is trying to reach a goal somewhere else but it cannot

see the path to get there. “We could have a helicopter fly over, build a three-dimensional map of the area, and send it back to the robot on the ground,” Coates describes. The robot on the ground could then figure out how to get to its goal from that data. The breadth of the applications of these learning algorithms is only limited by our willingness to teach our skills to computers. This research demonstrates that computers can learn to carry out extremely difficult “thought” processes, so it is possible that nearly computers will eventually be able to mimic anything humans can accomplish.

The Future of Computer Learning

While this project has reached (and, in fact, surpassed) its goals, the field of computer learning is still young and there is much to be done. It is not difficult to imagine several scenarios where machine learning would be beneficial. Robotics, ground and air vehicle control, and manufacturing, among others, are suitable areas for applying learning algorithms that would be greatly transformed by such concepts. However, one important question remains: how far will we go in granting computers the ability to think, learn and perhaps eventually feel? At this point, since we are far from mimicking human abilities, such considerations are not paramount, and research continues unabated. In the future, as these algorithms become more complex and gain broader abilities, we may need to be more cautious with our research. As for now, we can commend the magnificence of the work of Ng’s team and eagerly wait for what is next.

To Learn More

For more information, visit the website of Stanford Autonomous Helicopter at heli.stanford.edu

Credit: sxc.hu

The helicopter learns how to recover its original position after flying upsidedown.