The Next Photography Revolution-discovery December 2002

The Next Photography Revolution Here comes a digital-camera chip that could change everything By Eric Levin Photography by Tom Tavee DISCOVER Vol. 23 No. 12 | December 2002

"It's easy to have a complicated idea," Carver Mead used to tell his students at Caltech. "It's very, very hard to have a simple idea." The genius of Carver Mead is that over the past 40 years, he has had many simple ideas. More than 50 of them have been granted patents, and many involved him in the start-up of at least 20 companies, including Intel. Without the special transistors he invented, cell phones, fiber-optic networks, and satellite communications would not be ubiquitous. Last year, high-tech high priest George Gilder called him "the most important practical scientist of the late 20th century." "Nobody," Bill Gates once said, "ignores Carver Mead."

Digital cameras have relied on image sensors that can't do what color film does: record all three  primary colors of light at each point in the image. Instead, each light­sensitive point in the sensor  measures just one color—blue, green, or red—and complicated software in the camera calculates the  missing colors. Foveon's breakthrough X3 chip solves the problem with a three­layer design that  captures red, blue, and green light at each point. To demonstrate quality differences, the monarch  butterfly on this page was photographed with three cameras: an $1,800 Sigma SD9 with an X3 chip; a  $300 Nikon Coolpix 2500; and a $2,300 Nikon 35 mm F5 film camera. Insets show magnified detail from  each camera's image.

And now one of Mead's simplest ideas—a digital camera should see color the way the human eye does—is poised to change everything about photography. Its first embodiment is a sensor—called the X3—that produces images as good as or better than what can be achieved with film. That would make the X3 the most important advance in photography in nearly 70 years, but the long-term implications are even richer. In a year or two, you will be able to pack a true hybrid camera on vacation. It will take high-resolution stills, or upon the flip of a switch, it will take fulllength, full-motion video far exceeding the capabilities of present-day hybrid cameras. In the long run, X3 technology could even make cell-phone video sharp enough to project onto a big-screen TV, which would make dandy travelogues to send back to the folks at home, or enhance collision-avoidance systems in automobiles, or improve robot vision. X3 is the latest and most innovative product from Foveon Inc., the Silicon Valley digital-imaging company that Mead, 68, founded in 1997. Named for the fovea centralis—the part of the human retina where vision is sharpest and most color perception is located—Foveon took as its mission another radically simple idea Mead loves: "Use all the light." Don't cameras already use all the light that enters the lens? Film cameras do, but digital cameras, with few exceptions, don't. As Mead puts it, "They throw away two-thirds of the light." That makes sense only if you understand how a typical image sensor works. It's basically a rectangle of silicon on which millions of microscopic

light-sensitive pixels (technically they're not pixels, but that's what these light-sensing points have come to be called in the digital-camera business) are arranged in a grid. Pixels can't sense color. So a checkerboard of tiny red, green, or blue filters must be bonded to the surface of the sensor so that each pixel lets in one of the three primary colors of light. In so doing, it blocks out the other two. By comparing each pixel's single-color reading with that of its neighbors, software can derive the values of the two missing colors at each site. That takes approximately 100 calculations per pixel. In a four-mega-pixel camera, a size commonly available today, that adds up to a lot of number crunching. The process is called interpolation, and Mead has a less kind name for it. "It's a hack," he says. "They have to do all this guesswork to figure out what they threw away. They end up with a lot of data, but two-thirds of it is made up. We end up with the same amount of data, except ours is real." That is because X3 does what until now only film has been able to do: in one exposure, on one image plane, measure all three primary colors of light at every point on the picture. By doing so, it does away with the bugaboo of so-called mosaic sensors, which often guess wrong, especially at the edges of complicated patterns, introducing moiré effects and jagged color errors called artifacts. Sensing all three colors at each pixel sounds simple, but more than one industry analyst has described it as "the holy grail" of digital photography. "Engineers have been trying to solve this since the earliest days of digital imaging," says Alexis Gerard, publisher of The Future Image Report. Phil Askey, whose exacting equipment tests on his Web site, dpreview.com, are must reading in the trade, says, "This could be the first sensor to truly surpass film." The only camera to contain an X3 sensor now is called the Sigma SD9, a single-lens reflex with a price tag of $1,800 (not including the lens). But about this time next year, point-and-shoot cameras should be available from other manufacturers with X3 technology. They will have chips with slightly less than half as many pixels as the chip in the Sigma and sell for about $500. To be sure, Foveon will not find it easy to elbow its way into a market heavily committed to existing technology. But it has some influential advocates, including Microsoft's Gates. X3 is based on a well-known property of silicon: It absorbs different wavelengths, or colors, of light at different depths from its surface. A standard wafer of pure crystal silicon—the polished disc, five to eight inches in diameter, on which most microchips are made—is about 1/25 of an inch thick. The absorption of visible light takes place within 1/10,000 of an inch of the surface. If you think of that 1/25 of an inch of silicon as if it were a place in the ocean where the water is 1,000 feet deep, then all the light absorption would be taking place within two or three feet of the surface. At that scale, a human hair would be about 50 feet thick. What Foveon has done is imbed a sandwich of three light sensors within that first 1/10,000 of an inch. How they do it is a guarded trade secret, but the principle is basic physics. Blue light, which has the shortest visible wavelength, about 1/50,000 of an inch, is absorbed closest to the surface. Green light, which has a longer wavelength, penetrates a little deeper. Red light, with the longest wavelength, about 3/100,000 of an inch, burrows down farther before it is absorbed. As the photons strike the silicon atoms, electrons are released. These create electrical charges that the sensors measure. It took film almost a century to figure out the best way to do color. But once Carver Mead says, "The eye itself has  Kodachrome was perfected in 1935, competing schemes largely faded away. By taught us that remarkable things can  devising the simplest and most reliable solution to the problem—a three-layer be accomplished by building  emulsion—Kodak won the color war. Since then, color film has undergone intelligence into the image plane. An  many refinements, and other companies have grabbed significant market share. intelligent image plane gives you  But the three-layer emulsion is still gospel. higher quality photography with less  "Exactly the same thing is going to happen with electronic capture," says demanded of the user." Mead. "X3 is going to be the surviving image-capture technology. There's no question about it." If the rest of the industry isn't quite ready to throw in the towel, it's because the technology that Mead is up against has been king of the hill, despite all its flaws, almost since it was invented. If you have ever bought a video or digital camera, you have probably seen the letters CCD stamped on the box. They stand for charge-coupled device. Versions of the CCD were invented independently at Bell Labs and Philips Electronics about 33 years ago. They produced much better images than could be obtained with other kinds of solidstate sensors, and they took off. Specialized long-exposure CCDs have long proved invaluable in astronomy, and the

CCD put video cameras and digital still cameras under countless Christmas trees. If the CCD was the hare of the digital-imaging race, something called CMOS, for complementary metal-oxide semiconductor, was the tortoise. CMOS is a sophisticated process developed in the 1960s that produces chips with many transistors. Without it, X3 would not have happened. Although CMOS chips didn't make good images at first, they made terrific microcircuits and became the backbone of the computer revolution. CMOS was the technology Gordon Moore, the first president of Intel, had in mind when he made his famous 1965 prediction—subsequently known as Moore's law—that transistor density on a chip would double every year. Before he refined his prediction, changing it to doubling every 18 months, Moore consulted with an expert in circuit miniaturization—brilliant young Caltech professor Carver Mead. Mead has been fascinated by electricity since he was a child in Big Creek, California. His father worked in a hydroelectric plant that supplied power to Los Angeles, so Mead got to watch as new generators were brought online. He was awestruck by the immensity of the machinery and the force that turned it. He earned his bachelor's, master's, and Ph.D. degrees at Caltech, and became a renowned professor of engineering and applied science. With Caltech's Richard Feynman, the Nobel laureate, and biophysicist John Hopfield, he designed a course in 1986 called The Physics of Computation, which became an instant classic, and its principles are still taught by its creators' academic descendants. In 1969, Mead took Moore's law to a new level. Miniaturization, he said, would make it possible to build very-large-scale integrated (VLSI) circuits. They weren't large in size—in fact, they became smaller and smaller—but they incorporated more and more transistors on a single chip as the transistors themselves shrank. Mead predicted that transistors would eventually get to be as small as six-millionths of an inch across, or .15 micron. (Coincidentally, that is the size of the transistors in today's state-of-the-art Pentium 4 microprocessor, made by Intel. Foveon's X3 is right behind it, at .18 micron—the smallest transistor in an image sensor.) Mead laid out an innovative set of concepts for designing VLSI circuits, which became the industry standard as his students went out into the work world. Dick Lyon, now Foveon's chief scientist, was one of those students. "Carver taught a generation of engineers how to cope with the complexities of millions of transistors on a chip before anyone believed there would ever be millions of transistors on a chip," Lyon says. Most sensors in point­and­shoot  In the 1980s, Mead turned his attention to an even more complex and digital cameras are smaller than a  ingenious kind of circuitry: the human nervous system. It is clearly the most successful computing system of all time, and it is an analog, rather than a digital, fingernail, but the X3 chip is closer to  system. That part intrigued Mead the most. Instead of registering information in the size of a frame of 35 mm film. So  are the sensors in other top­of­the­ digital strings of ones and zeros, an analog system, such as a retina or an ear, line digitals, but the X3 is complex to  measures a continuum of values. Digital systems in the technological world were quick and complex; analog systems were slow and simple. Mead thought manufacture because of its three  layers and its transistor density. "If  he could learn a lot by studying—and attempting to model in silicon—the you were to buy a Foveon chip," says  natural masterpieces of human analog design. This was such a new field, it had no name. So he gave it one: neuromorphic Chris Joyce, director of process  technology at National  electronics. And it grew into businesses. His study of the cochlea was Semiconductor, "you could  instrumental in the founding of Sonic Innovations, a digital hearing aid company. Synaptics, which he cofounded in 1986, created the laptop TouchPad. deprocess it and figure out what  we've done. But you wouldn't be able  Foveon was spun off from Synaptics and National Semiconductor, a chip manufacturer, to pursue the ideal of a digital image sensor that would see in full to figure out how we've done it." color, like the human eye. A slender, energetic man with a trim Vandyke beard, Mead has a disarming, gentle speaking voice and a face that recalls both David Carradine in Kung Fu and Richard Kiley in Man of La Mancha. He rarely raises his voice, but when something gets under his skin, you'll know it. "The more I learned about human vision," he says, "the more it was clear that what these mosaic sensors were doing was introducing artifacts into the image. It was one of those things that becomes so massively annoying that after a while you think you ought to go do something about it. It was clear that the way image sensors worked was brain-dead. I talked to a lot of people, and nobody got it. So I finally said: 'That's not a problem. That's an opportunity.'" Among Mead's many talents is putting together a brain trust. From National Semiconductor, he recruited an iconoclastic and fertile-minded inventor and electrical engineer named Dick Merrill to be his design wizard. His

choice for chief scientist, Dick Lyon, had worked with him on several neuromorphic projects. Mead's artificial-retina project at Caltech had benefited greatly from ideas Lyon had developed in inventing the optical mouse at Xerox's Palo Alto Research Center in 1980. At the outset, X3 wasn't even a pipe dream. Foveon's first cameras were based on patented prisms that split incoming light into its primary colors, directing the light into three separate sensors. The system produced extraordinary images but was cumbersome (the camera had to be attached to a laptop) and costly. So one of Lyon's first tasks was to sift through a pile of ideas that Merrill had written and see if any of them seemed promising. One did. It proposed a way to build a sensor by taking advantage of the light-absorbing qualities of silicon. Merrill had hit upon this method by accident. Back at National, he had built a sensor that didn't separate color the way he expected it to. "So, not knowing much about sensors, I decided to look into why that was the case," Merrill recalls. Then he pretty much forgot about it, until the day Lyon came by asking if he really thought it could work. Merrill thought it could, so they took it to Mead, who was skeptical. It turned out Mead had experimented with exactly that concept at Caltech and found the color separation that silicon achieved was too indistinct to be useful for photography. Merrill and Lyon "massaged it around," as Lyon says, and brought a revised design back to Mead, who was encouraged. They conducted simulations, but the color separation was still fuzzy. "We were tearing our hair out about where to go with that," Lyon recalls, "when Merrill came up with a much more radical structure for how to make the thing." "Once we worked through the science of it, and Merrill and Lyon convinced me it was fundamentally sound, I've never doubted," Mead says. "We basically bet the company on it." Now the ball was in Lyon's court. When he got to Foveon, he realized that the one expertise the company was missing was optics and color theory. "So I basically taught myself from books what I needed to know," he says. Lyon's colleagues marvel at his polymathic abilities, not to mention his large collection of slide rules, including a working seven-foot-long Pickett that he hung above the bookshelves in the company library. Lyon believed he could find a way to use the soft color separation in silicon advantageously. The reason he thought so, as he reminded the others, is that "the eye itself does it that way. The color sensitivities of the cones in the retina are not sharply defined. On a graph, they would look like smooth overlapping curves." In clean­room bunny suits, chief 

scientist Dick Lyon (left) and chief 

In Merrill's new architecture, the top, or blue, sensor would pick up a little green designer Dick Merrill play with early  and red light as those rays of light passed through. The middle, or green, sensor Foveon cameras. Merrill sees his job  would pick up the leftover blue and a little red, and the bottom, or red, sensor as "making a list of all the reasons  would pick up the leftover green and the last vestiges of blue. All digital people don't already own a digital  cameras transform the raw measurements that come off the sensor into a set of camera and knocking them down one  numbers called a matrix, which allows a computer monitor to display them as by one." images. The X3 needed much bigger matrix numbers than most sensors, and that made highlights and shadow areas harder to render. "Even with film, it's a challenge to produce good color in highlights and shadows," Lyon says. "Professional photographers know how to light a scene to work around that. But the issue for us was getting good pictures when shot by amateurs, where the lighting is really challenging. That's what kept me up at night." Mead's goal of "making the image plane more intelligent" hinged on more than measuring all the colors. The vision was of a sensor that could be used to create video as well as stills—and, unlike current cameras, do both well —and eventually take on tasks such as focus and exposure, now handled by expensive supplementary chips. There is no way a CCD can do all that. It has its hands full merely schlepping the electrical charges from pixel to pixel in a process that is usually described as a "bucket brigade." By applying a positive voltage to the negatively charged electrons in the pixel, the CCD attracts the electrons and essentially hands them off from one pixel to the next. There are different patterns, but typically all the pixels in the bottom row of the array are shuttled off, then the charges from the line above drop down, something like a row of blocks in a game of Tetris. It's not only a relatively slow process, but applying all those successive voltages also makes CCDs power hogs, which is one reason users of digital cameras have to replace or recharge their batteries so often.

So this is where the tortoise re-enters the story. Only about half the surface area of any given chip is actually devoted to light sensing. The rest contains supporting circuitry, which in a CCD mans the bucket brigade. By using state-of-the-art CMOS technology, Foveon was able to pack more transistors into each pixel, which enables the chip to do more things. Equally important, CMOS architecture allows the camera software to communicate directly with the pixels and read them out in parallel—meaning all at once—rather than serially, one after the other, as in a CCD. The greatest of all parallel processors is the human nervous system. The signals generated by the rods and cones in the eye's retina, for example, don't stand in line waiting to get into the optic nerve like cars backed up at a tunnel. "They're actually processed rather extensively right in the retina," says Mead. The brain samples the signals in parallel, taking information as it comes. "The nervous system always operates on partial information," Mead explained in an interview with the technical journal EE Times. "Its basic assumption is that you start with no information, and anything beyond that is something, and it is a heck of a lot better to have something rather than nothing. But what the digital paradigm is based on is, 'I've got to have all my arguments before I start any operation'—the opposite end of the spectrum!" The significance of X3 is that it advances the art of parallel processing and brings us closer to an age of truly smart machines. CCDs are dinosaurs, and it isn't only Carver Mead who thinks so. A 2002 report by the market research firm In-Stat/MDR projected that production of CMOS sensors would draw even with CCDs in 2004 and pull away thereafter. Much of the growth will be in the lower end of the This photograph, taken with the Sigma SD9  market—in cell phones, Webcams, and toys, which are already popular using a Foveon chip, attains levels of  in Japan. But Foveon was encouraged last year by Canon's introduction sharpness and color accuracy usually seen  only in medium­format cameras that use 120  of two high-end CMOS cameras, both of which use a filter mosaic. mm film, which has a resolution about twice  "They've made the world safe for CMOS," quips Lyon. that of 35 mm film. Dinosaurs, of course, ruled the earth for a long time. CCD makers have managed to develop a number of ingenious processing strategies to compensate for the system's inherent weaknesses. And they've managed to bring down prices steadily while increasing pixel count. Even though there's more to producing good images than stuffing greater numbers of smaller and smaller pixels into a camera, smart marketing has established the mega-pixel as the sexy equivalent of horsepower. Mead has another paradigm in mind: "Whenever a radically new technology has developed, a new major company has come out of it. When the transistor came along, we got Texas Instruments. When the integrated circuit came along, we got Intel. When we got microprocessors and personal computers, we got Microsoft. "That's the way I see Foveon. It doesn't mean we're going to put the others out of business. We have no intention of doing that. They're becoming our customers. We're forming alliances. "We're not going to be an Apple," he adds. "We're not going to turn ourselves into an island. We're going to be more like a Microsoft or an Intel." It's amazing, come to think of it, how much Mead does look like Don Quixote.