Dick Lyon (BS '74), Electrical Engineer and Pioneer of Optics and Hearing Accessibility
If you are reading this on a desktop computer, you are most likely navigating with a mouse. And if you are old enough to remember, the original computer mouse relied on a rolling ball. The mouse you are using is most certainly an optical mouse, which uses light (usually an LED) to track the movement of the computer user's hand, and which operates more accurately and without mechanical degradation compared to its predecessor. For this innovation, we have Caltech alumnus Dick Lyon to thank, who was one of two independent inventors of the optical mouse. Because of its ubiquity, this is arguably the most well-known of Lyon's innovations, and his list is as impressive as it is wide ranging.
In the discussion below, Lyon reflects on his upbringing in Texas and his undergraduate experience at Caltech, where Carver Mead was revolutionizing the world of digital design. In his career, Lyon has always straddled technology design and information accessibility in the pursuit of making electronics devices more efficient and more intuitive in their interactions with human senses. After a formative summer experience at Bell Labs, Lyon began graduate work at Stanford, ultimately leaving with a Master's degree, as he was eager to enter the burgeoning world of Silicon Valley. His long and illustrious career in the technology sector included appointments at Xerox PARC and Apple Computer, and he describes with fondness his reconnection with Mead and their work on VLSI, for which Lyon contributed important research in the area of signals processing, and for Foveon, which Lyon co-founded, based around the X3 sensor's unique manner of capturing digital images.
In more recent years, Lyon has worked as a research scientist at Google, and he explains what continues to make Google such an excellent center of research. His current interest is in the area of machine hearing, which uses signal processing to improve speech recognition technology. His book Human and Machine Hearing: Extracting Meaning from Sound is the standard reference on the subject, and he continues to refine this work since its publication in 2017. After experimenting with a soft retirement, working at half time, Lyon decided to bump his work time up to seventy percent. The reasons for this are obvious: Lyon is too committed, and the research is too interesting, to let go. And for this, Lyon credits his time at Caltech, where leadership in technology innovation rested on a foundation of creativity, hard work, and intellectual fearlessness.
Interview Transcript
DAVID ZIERLER: This is David Zierler, Director of the Caltech Heritage Project. It's Friday, August 25th, 2023. I'm delighted to be here with Richard F. Lyon. Dick, it's great to be with you. Thank you so much for joining me today.
RICHARD F. LYON: Thanks for inviting me, David. I look forward to our conversation.
ZIERLER: Terrific. Dick, to start, would you please tell me your current title and institutional affiliation?
LYON: What's my title? Some kind of research scientist. Principal Research Scientist at Google.
ZIERLER: What are you currently working on these days?
LYON: Mostly on what we call hearing accessibility – hearing aids, cochlear implants transcription devices, and stuff like that – and also application of some of the same sound analysis techniques to more conventional products, things like speech recognition.
ZIERLER: Dick, is this full-time work for you? Are you easing into retirement at this stage?
LYON: Yeah, a year ago when I was 70, I switched to 70% time. Actually, I switched to 50% time and then, after a little bit, I decided I couldn't work that little, so I switched it back to 70%, and that's where I've been for a while.
Google Greatness
ZIERLER: Dick, after your long and illustrious career, I'm just curious, what can be read into the fact that at this stage in your life, you want to work at Google? What does that say about you? What does that say about Google?
LYON: [laugh] Google is, in many ways, the best place I've ever worked, and I've been here about twice as long as anywhere I've worked before. I'm going on 18 years here, and my previous record was 9 years at Apple. It's just been a fantastic place. There's been a lot of grousing recently with Google growing up and acting more like a big company rather than the scrappy outfit it was when I joined. But it's still good. There's still the most amazing concentration of super smart people that I've ever had an opportunity to work with – and the perks are nice too.
ZIERLER: I'd like to ask a general question about applied research and basic research, and I think you're so well positioned to reflect on this because you have connections, of course, going way back to Bell Labs, which is considered the high church of fundamental research, not having anything necessarily to do with the company bottom line, to a place like Google, which obviously has lots of fundamental research. What does that look like for you? Where is the company these days in that regard?
LYON: Basically, my whole career has been more on the applied side, including the time at Bell Labs where I was working on applying digital signal processing theory to make devices that would lead to cost reductions in the telephone system. At Google, my tendency is to work on things that I think people will use. I've never really gravitated to pure research but, on the other hand, I've enjoyed working in environments where we were free to pursue our ideas, not necessarily connected to a product but with the idea that they'll lead to something important. At Google, traditionally, the way research is organized, and the way research decisions get made at Google, is we hire people that have research leadership skills, people who might otherwise go lead a group at a university or something like that, and we expect them to lead. We expect them to decide what to do. At the same time, we have a system of peer-based evaluations where people know they're going to be judged on having impact, and the easiest way to have impact at Google is to get your ideas to come out in a product. There's a lot of self-organization and motivation that leads people to choose to work on things that are going to matter to the end user at some point. It doesn't mean everyone's doing exactly that. There's still people that choose to do more fundamental stuff without any idea of where it's going to go. It's all tolerated. I'd say recently with some of the reorganizations, there's a lot more emphasis on kind of top-down strategic management and decision-making about what to work on, which is not super comfortable for everybody that's used to having the freedom to do their own thing all the time. But it's a mix. It's a balance.
ZIERLER: Dick, a question about intellectual property, both at Google and all of the places that you've worked. Obviously so many of the things that you work on are useful in the market for societal benefit. How does that work in terms of establishing a patent? Where are you working as an individual? Where are you working as an employee for these technology companies?
LYON: I've generally always been an employee, and I have a certain amount of respect for their intellectual property regimes. Companies' research organizations and otherwise that I've worked in have generally had some kind of an invention or patent incentive system. They incentivize you to write up what you've done that's new and possibly patentable, and then they might go and patent it. They give you a little bonus for doing that. The result of that is a lot of people write up invention proposals, many of which are just kind of crazy ideas designed to get a bonus. Perhaps I've done some of that, but I've invented some things, and I've got about 75 issued US patents along the way.
ZIERLER: Because you have such long perspective on this, where is the patent process today versus at the beginning of your career? More administrative? More bureaucratic?
LYON: No, I wouldn't say that. At Foveon, where I was for eight and a half years, one of my responsibilities was to be in charge of their intellectual property portfolio, and I worked closely with a patent lawyer there that taught me a whole lot of stuff about all that. We got a whole lot of patents there. None of them ended up being worth anything in the long run because the company didn't really take off, and didn't really need to protect their inventions. [laugh] But I came away with a bad attitude about patents. That is, at that time, I spent a lot of time and effort trying to get valid patents that properly disclosed how to practice the invention and so on, get strong patents, and I came to realize, and especially at Google, that a patent is almost never going to be challenged in court. It doesn't need to be any good. It's just a stack of paper that companies use to threaten each other or defend themselves from threats or whatever. My attitude toward patents now is let the lawyers write it the way they write it. It doesn't make a lot of sense. I'll read it and say, "Yeah, that fairly represents my invention." I'll sign off on anything. Someday, if some lawyer hears about that, I'll be in trouble.
ZIERLER: [laugh] Dick, some professional biography questions. We'll just start with titles and areas of expertise. Fundamentally, do you consider yourself an electrical engineer or a computer scientist, or are those distinctions really not particularly relevant for you?
LYON: They're no longer very relevant here, but I'm definitely an EE. I got my degrees in electrical engineering, and I never had a computer science class. But, on the other hand, I learned early on – I was a hardware guy doing hardware for digital signal processing and things like that – I learned early on that most hardware projects are mostly software, so I do a lot of that too. [laugh]
ZIERLER: What does that mean? I wonder if you could unravel that a bit.
LYON: When we were designing camera systems for Street View, there's some optics and image sensors and electronics and all the hardware and all that stuff. But if you look at the scope of the project and the people on it, you'll see that it's mostly software engineers that have to take what that hardware's doing, and make a system out of it. It was the same way at Foveon, where we were making cameras and image sensors, and I had to design the image processing pipeline and the way the software works and all that stuff. I ended up semi-managing the software too.
ZIERLER: Dick, are there any theories that provide an intellectual basis for what you might consider your inventing work, your experimental work?
LYON: There have been some themes. Some number of my inventions have to do with human perception. I got into that early on, first of all, being exposed to the telephone system at Bell Labs, where a lot of what they do was based on a long history of studying how people perceive sound. Then later, when I got into speech recognition, I decided that in order to make it work better, we really needed to understand how hearing works. I got into stuff about how the visual system works when I did the optical mouse design at Xerox. Similarly, in handwriting recognition at Apple, a lot of that was visually oriented and brain-model oriented. It's a lot of influence from thinking about how does human perception work? At Google, I'm in a group called Perception. When I was at Apple, I led a group called Perception Systems for a few years. It's a thread that has run through most of the outfits I've worked for.
Silicon Valley Without the Silicon
ZIERLER: I think a question in historical perspective, you've been at Silicon Valley since its inception, since its early maturity, however you want to define those things. Where are we now, circa 2023? Does it feel still like the golden years? Is it middle age? Is there an exciting future, 25, 50 years out? What does that look like for you?
LYON: [laugh] It doesn't even feel like Silicon Valley anymore. There's not that much silicon work around here. It's more like Software Valley. It's all the big software companies.
ZIERLER: This is to say Silicon Valley is now really Taiwan?
LYON: [laugh] Taiwan is where a lot of the stuff gets made. Designs can be done anywhere. You don't need to be near where silicon gets made to design chips. The design is everywhere, and the manufacturing is mostly at TSMC in Taiwan. [laugh] But the Valley itself, it's been a marvelous place. The growth has positive and negative sides, and the traffic's gotten a lot worse over the years, and all the little orchards have disappeared – apricot orchards and the cherry orchards and so on that were still here in some abundance when I got here in '74. But I watched a video interview with Frederick Emmons Terman, who was the head of EE at Stanford for many years. They interviewed him in 1968, and he was talking about how the Valley had changed. He also had this balanced view, like, he missed the orchards going away, but he thought the growth of the high-tech companies was a very good thing. We've made a lot of billionaires here – I'm not one of them – but [laugh] it's a pretty nice area in a lot of ways.
ZIERLER: Dick, for all of the technological advances that you've made possible, what have been some technological advances that you weren't part of but were really game-changers for the kinds of things you're able to do?
LYON: Oh, there were so many. I watched people all around me over the years taking off and doing things, where I'm thinking, "Yeah, that looks boring," and it turns out to be a big hit. Back around 1980 or maybe '82—I forget exactly when—my little brother Tom was talking about joining this little start-up called Sun Microsystems. I said, "The world doesn't need another Unix workstation."
ZIERLER: [laugh]
LYON: That took off, he did well, and my brother Bob joined him as well, and they both did very well. My friend at Stanford, Andy Bechtolsheim started it. Jim Clark, another professor at Stanford at the time, said he could get me like a half percent of the company he was starting, which was Silicon Graphics Incorporated. I thought, "I don't want to be working for a computer company." To me, a computer was a tool to do interesting stuff. I didn't want to be in the computer industry. That kind of changed when I went to Apple. [laugh]
ZIERLER: For some of the misses that you've had, has your sense of intuition, would you say, has it improved over the years? Do you have a better batting average now in thinking about what's viable and what's not?
LYON: No. I'm a terrible futurist.
ZIERLER: [laugh] Hasn't changed?
LYON: I think I really missed the AI revolution while sitting here in the middle of it. [laugh] To me, the emphasis on big data and machine learning, yeah, it's important. You can do good stuff that way. But I'm more attuned to the small data stuff, I think, and wanting to help people with their hearing and stuff like that, which does leverage some of the big data things like improved speech recognition. But my tendency has been to work with my EE background, and try to improve the front ends of these things. I've done some good for products that way. But the big movements have been in the other end, the big data end of things. That hasn't been really what interested me.
ZIERLER: Dick, let's go all the way back to the beginning now, back to your childhood in Texas. Let's start with your parents. Are they also from Texas?
LYON: Yeah. My dad and I both were born and raised in El Paso, Texas. My mom was born in South Texas, little town called Poth, from German-speaking immigrants. We go way back in Texas. On my dad's mother's side, her mother was in East Texas, and had a great-grandfather there who died of his Civil War wounds, and had a great-great-grandfather on my dad's father's side who was a Southerner in Georgia, who was an associate justice of the Supreme Court of Georgia before, during, and after the Civil War. That's the guy I was named after, Judge Richard F. Lyon. He had some juniors and thirds and so on, but I wasn't quite in that straight line.
ZIERLER: Dick, what was your dad's profession?
LYON: He was an electrical engineer also.
ZIERLER: Aha, family business.
LYON: He spent his entire career at the El Paso Electric Company.
ZIERLER: Did he involve you in his work? Did you ever get to see what he did, what it meant to be an electrical engineer?
LYON: Yeah, a little bit. When I was in college, I did some little projects for him. He was interested in trying out some stuff, and I designed some circuit boards, and made them for him and stuff like that.
ZIERLER: What about your mom? Did she work outside the house?
LYON: Mostly not. She did substitute teaching at the school for a while. She had been a journalist before they got married and moved to her El Paso.
ZIERLER: Growing up, were you always on the science and math side of things? Was that always your interest?
LYON: Yeah, definitely. We were tinkerers. As a kid, I'd get graph paper and I would graph curves. I was fascinated by parabolas when I was barely old enough to multiply numbers together. [laugh] I'd fiddle with the math, and I'd make things. My dad was always making things too. He had all kinds of machine tools in the garage. He had a lathe, a really nice metal lathe. He had nine kids. Of the lot, I was the only one allowed to use the lathe without him looking over my shoulder.
ZIERLER: Wow, high praise. [laugh]
LYON: [laugh] Yeah. We had a dark room. He had a dark room, and we'd develop his film, and print his pictures. I got into photography, and learned how to shoot some of his old cameras and so on.
ZIERLER: Public school throughout in El Paso?
LYON: No. My mom was Catholic, so we all went to Catholic grade school, and then switched into the public high schools. The public schools were pretty good, and the Catholic school was good too.
ZIERLER: Dick, how did you hear about Caltech? Did you know names like Dick Feynman and Linus Pauling when you were a kid?
LYON: No, not at all. I knew very little about Caltech. One of my teachers when I was in grade school mentioned that I was smart – maybe I'd go to MIT. I said, "Huh, what's that?" "Massachusetts Institute of Technology." Then I noticed at some point there's also a California Institute of Technology, so I figured that must be just as good. [laugh]
ZIERLER: [laugh]
LYON: Literally, I knew nothing. My older brother, who I used to fight with a lot, he went to Princeton. I just had to look at the map, and see how far away from him could I get.
ZIERLER: [laugh]
LYON: At some point, I applied Caltech, MIT, and Stanford, and they all sent, well, I don't know if they all sent interviewers. Caltech sent an interviewer, so I got to talk to someone. They all accepted me, and I chose Caltech.
ZIERLER: Did you graduate at the top of your class?
LYON: From high school? I wouldn't say top. I don't remember any particular honors there, valedictorian or anything. But we had a strong class; a lot of really smart people. Some went to Yale and Princeton and stuff like that from the public schools. I was, I guess, more or less a straight A student, but top of the class was maybe straight A-pluses or something.
ZIERLER: Dick, tell me about arriving at the Caltech campus, your first days there. What was that like?
LYON: Oh my god, it was horrible. First of all, one of the reasons I chose Caltech too was it was within driving distance from El Paso. El Paso's just halfway between Pasadena, California, and Pasadena, Texas, believe it or not. Only 800 and some miles either way. My grandparents decided they would drive me out to Caltech, and my dad decided he would come along. There's four of us in my granddad's Cadillac. Spent the night in Tucson, and the next day on the road, I got sick as a dog, and so my arrival at Caltech, I was very sick. What I remember was the drive in there, how dirty the air was. You couldn't see anything. It's just smoke everywhere. This was peak smog, 1970. It took me a long time to get over that. Unbeknownst to my parents, I had gone off to Juárez, Mexico, the night before the trip, and ate some stuff that probably gave me Montezuma's revenge.
ZIERLER: [laugh]
LYON: My grandma says, "Oh, it must have been those pancakes we ate in Tucson." I said, "Yeah, it must have been."
ZIERLER: [laugh]
LYON: Anyway, I was in bad shape when I started there, and it took months really for me to get over that. But the other thing I remember when I started there, it was three weeks in when I was walking down the Olive Walk one day, and I saw there are mountains there.
ZIERLER: Right.
LYON: Three weeks, I had not known there were mountains there.
ZIERLER: Right. That's how bad the smog was.
LYON: So smoggy.
ZIERLER: Dick, did it register with you at the time—if I have my chronology right, you started at Caltech the year that they went coed.
LYON: That's right.
ZIERLER: Did you know that at the time? Did you know that you were part of that inaugural class?
LYON: Yeah, I don't know that I knew it ahead of time, but I certainly was aware of it as a freshman. The other thing they did my year, they had a tradition of doing freshman orientation on Catalina Island, which was really nice. Except my year, they decided not to do that. They had it on campus, and that was such a disaster that they went back to doing Catalina Island. I had the one year that missed that apparently wonderful experience.
ZIERLER: Dick, once you got your bearings in class, did you feel well prepared coming from El Paso, relative to your classmates?
LYON: Yeah, I did. I didn't find the classwork or homework or anything to be super difficult. I just did it. Some people were struggling a lot, especially with the Feynman physics class where we would read Feynman, and then go home with problems to solve. A lot of people couldn't do that. They couldn't connect what they had read in the book with the problems that we were trying to solve. I ended up helping a few of them with that. But I had somehow been more able to do that. I had a pretty good background in the electrical arts already, so the EE classes were easy.
ZIERLER: Now, did you come in with the game plan to pursue electrical engineering, or you developed that in real time?
LYON: I don't think I was committed, but I think that's kind of where I was. I did a lot of electrical projects as a kid, and ended up signing up for a number of EE classes to learn more about that.
Digital Design at Caltech
ZIERLER: What was exciting as a freshman for you in electrical engineering? What seemed like the frontier?
LYON: Basically digital design. The course that Carver Mead taught—I think it was called EE 4. Maybe it was EE 3. I forget which one it was. Both of those were great courses. One of them was on how electronic devices work, and the other one was how to design a digital system out of medium-scale integrated circuits. Carver taught that latter one, and I think maybe Humphrey taught the first one. I forget, but amazing courses. These low-numbered courses that are taught to freshmen were things that at most schools you'd probably have to be a graduate student to get into at this time. It was super exciting. I didn't get into chip design until a few years later. Carver was already starting into that. I'd done a little bit by that time, but I didn't really get into it. I had a peculiar relationship with Carver, too.
ZIERLER: How so?
LYON: He doesn't remember this aspect of our relationship. He was my undergraduate advisor. He was rather aloof. He was always busy, and didn't have a lot of time for undergrads. He was advising [laugh] so I'd go in there with my card for him to sign. "Here's the courses I want to take." He would take one glance, and sign it, and send me on my way. [laugh] He doesn't remember meeting me until some years later. [laugh] He told me a story about what he remembers from meeting me, which is when he was teaching a digital logic lab class, where you had to build a project that works. That was basically the only criterion for passing is that you've got to build something and it has to work. You have to demonstrate it. I built a little digital tachometer for the junker of a car that my roommate and I had bought.
When it was time to show it, he's in the lab with all his students. He calls out, "Lyon?" I said, "I'm here." He says, "Where's your project?" "It's upstairs in the parking lot." "OK, let's go see." [laugh] We ran up there, and I showed him my digital tachometer. That wouldn't have been enough for him to remember me. What comes next is he notices the numbers. The thing's idling, and the RPM number's not steady. It's flickering between different values. He said, "How come that's not a constant RPM there?" I said, "I worried about that a lot, and I studied the problem, and I realized this is a V6 engine, and the spark firings are not equally spaced. Depending on when you count the sparks, you might get a different number." He was blown away by that answer because I had actually thought about it, and that's what he remembers. That's when he remembers meeting me. [laugh]
ZIERLER: I wonder if you can talk through what was so mind-blowing about that achievement.
LYON: No, it wasn't an achievement. I just built a little counter circuit that worked. It was a trivial thing. But what he was blown away by was the fact that I had looked into why it was working the way it was, and I had learned something about engines, something peculiar to V6 engines. [laugh] I didn't do the bare minimum. I went a little bit for further, and that's what impressed him.
ZIERLER: Dick, what other EE professors did you work with at Caltech?
LYON: A couple. Hardy Martel and John Pierce had some projects that I worked on in the lab in the basement, 35 Steele basement room, like a project to send data over the wiring in a house, things like that. Hardy was a friend. He was a great engineering professor. I don't know if you've been around Caltech long enough to know these guys.
ZIERLER: I know of them. I know the names.
LYON: Charlie Wilts—or Chuck Wilts, as we called him—I didn't work with him in EE so much, but he was my PE instructor, that is, I got PE credit for taking his rock climbing class—
ZIERLER: Ah. [laugh]
LYON: —where I as a freshman, a third-quarter freshman, third term freshman year, I earned the distinction of being the only student in rock climbing to ever get injured in class. [laugh]
ZIERLER: [laugh]
LYON: I fell off a rock doing an unroped bouldering move off the ground, and tore my MCL pretty bad. He gave me an incomplete in the class, because to complete the class, you had to complete a multi-pitch leader climb on a big rock. I did that – the next summer, after I recovered, I went out with the climbing club to complete the class to get my PE credit. I did a really cool climb called Graham Crackers on Suicide Rock. It's got this overhanging block at the top you have to get over, and I managed it, and so I completed the class. He says to me at the top, "Well done. My daughter did that last week with casts on both of her legs."
ZIERLER: [laugh]
LYON: I'm thinking, "OK. I get it. I'm not a great rock climber like some families."
ZIERLER: [laugh] Dick, tell me about the student workshops at Caltech. What was available? Could you go in any time? Could you build anything you wanted?
LYON: There were different things. Under the Winnett Student Center, there was a shop with a lathe and a mill and a few things like that. I think you had to apply to get permission and a key or something like that. They didn't just let anybody in there, but you had to convince them you knew what you were doing. I'd go in there, and make things sometimes. But the EE department, they had what they call the EE Stockroom run by Jim Campbell, where you could get parts to build things. It wasn't free stuff, but it was low cost. It was a neat hybrid little store because they sold not only everything you needed to build your projects, but they also sold all the rock climbing supplies, the carabiners and ropes and stuff like that [laugh], because it was part of the department. It was a very cool shop. I'd hang out in there. Jim Campbell, I don't think he's still around there either. I think he's retired. But he would build and maintain the demos for Carver Mead's class on digital design. He was Carver's right-hand man in a lot of stuff. I got to know him real well. He was also Chuck's right-hand man in the rock climbing. We had a great relationship. I was always tinkering. A lot of the electronics tinkering I would do in my room at the dorms in the house too. I used to make printed circuit boards. I would make my own patterns with tape on transparent plastic and photo resist, and developing the photo-resistant with trichloroethylene in my dorm room, my house room, pouring that TCE down the drain – but that was back before it was a carcinogen. [laugh]
ZIERLER: Of course. [laugh] What house did you live in?
LYON: I was in Lloyd House.
ZIERLER: What was Lloyd House known for? What was its identity on campus?
LYON: It was a weird place. It was very fractious, I would say. There's two major camps, sort of like the way the country is polarized these days. We had the jocks and the LSD acid heads. [laugh] I wasn't in either of these camps. But there were some of us in the middle, and there were some people on both ends. I had a good friend who was a big druggie and a football player. It was a weird place.
ZIERLER: [laugh] Was the campus political at all, this being during the bombing of Cambodia, the Civil Rights movement? Did you feel that at all on campus?
LYON: A little bit. It was also while I was there that the military draft switched from having student deferments to having a lottery system. As an undergraduate, I got a number that said how far down the line I might get drafted. I was lucky. With the number 305, I didn't feel like I was under much threat because there was a number per day of the year – 365 is where you wanted to be. [laugh] That made things more political because people felt a lot more under threat. I don't know of anyone that got drafted out of Caltech, but it could have happened. There was one point where some students got up on top of the Millikan Library. What do they call it now? Caltech Center or something.
ZIERLER: Caltech Hall.
LYON: Caltech Hall. They got up on top of the Millikan Library, and rappelled over the side, and mounted a big sign that said "Impeach Nixon." That made the press. One of the big donors called up and said, "This is not acceptable." [laugh] They were going to try to punish Caltech if they don't punish the students that did that or something. I don't know what came of that, but there was some consternation around that whole politicization of the campus.
ZIERLER: Dick, what did you do during the summers? Did you stay on campus? Did you go home? Were there internships to pursue?
LYON: Yes and yes. I mostly stayed on campus. When I came to Caltech, I had spent the summer before my freshman year working at Texas A&M University, where I had good luck to get into a deal with a civil engineering professor there that wanted some computer programming help. In order to live on campus in their dormitories, I had to be a student. I signed up as an Aggie, and I took some calculus class over the summer. Yeah, I'm a Texas Aggie – they say once an Aggie, always an Aggie – that first college I ever enrolled in, so there was a summer job. The guy said he would have me back after my freshman year. But when I wrote him a letter at some point asking for details, "When should I show up?" he says, "Oh, sorry, I left Texas A&M, and now I'm [??]." I had to start scrambling for a job. I think that's when I talked with Hardy Martel, and he said, "There's this new guy here that could maybe use some help, named John Pierce." I said, "OK, I'll talk to him." I went and talked to John Pierce, and he put me to work in his lab. I'm not even sure I've got the chronology right. That was either after my freshman year or my sophomore year. The other summer deal that I can't remember which year was which, was I worked at the Jet Propulsion Laboratory for Wally Rippel, who's the guy who had won the first transcontinental electric car race against MIT.
ZIERLER: Yes.
LYON: I had a funny relationship with Wally too. He was a great guy. When I was a freshman, they had this deal for Thanksgivings where they would get families to host students. We piled a few of us in somebody's old junker car, and went off to Hollywood to the home of Wally Rippel Sr., who was a sound recording engineer in Hollywood. His son Wally Rippel Jr., the electric car guy was there, and the rest of their family and so on. We had a grand old time. Then later when I interviewed with Wally for the job at JPL, I got the job, and it was fine, and so on. At some point after I started working for him, and was talking about things, like his house or his family or something, he says, "How do you know my house?"
He didn't realize I was the same guy he had met back at Thanksgiving. It was funny to see him putting the pieces together. [laugh] But we worked on and off together for a couple years. I had summer jobs in the lab at Caltech. I had Jet Propulsion Laboratory. Then after my junior year, John Pierce got me a job at Bell Labs, what you would call an internship these days, but we just called it a summer job back then. Pierce had told them they should look at hiring me, and I had to fill out their application, and send in my transcript and all that. Then he calls me into his office one day, and he says, "Look at this." These guys said, they wrote back, and told Pierce, "Why should we hire this guy? He's got less than an A in some important subjects," because I didn't have all A's at Caltech. [laugh] Pierce just says, "I'll fix it." He just wrote them back. He said, "Hire him anyway."
ZIERLER: [laugh]
LYON: He used to be their boss, so they were used to listening to his judgment, and so he managed to get me this job, and it was fantastic. That was where I learned about digital signal processing.
Signal Processing at Bell Labs
ZIERLER: What group did you join?
LYON: It was digital signal processing research. It was, I think, the only research department that was located at Holmdel, as opposed to most of them, which were in Murray Hill, New Jersey. They were building digital signal processing based telephone equipment, like the transcoder that would convert telephone channels between the frequency-division microwave carriers, and the time-division digital representations that they needed in their switching system. These were big, bulky, expensive things, and they were looking at could they do it cheaper digitally. I learned all the theory and practice, and met a bunch of great people, and got to build things and write papers. I got a patent and a publication out of that summer, so it was a great experience. Then I came back to Caltech, and I taught Caltech's first course on digital signal processing.
ZIERLER: As a senior?
LYON: Yeah.
ZIERLER: Wow.
LYON: I had three faculty members taking the course from me.
ZIERLER: [laugh]
LYON: That was pretty awesome.
ZIERLER: Was that ever done before? Had you ever heard of students teaching classes?
LYON: It didn't seem unusual at the time. I don't know. I'm not sure.
ZIERLER: Did you talk to anybody at Bell Labs about taking a job there after Caltech? Were you considering that?
LYON: No. I didn't like living in New Jersey. I didn't have a good living situation there. The hot, humid summer made a negative impression on me, so I was happy to never think about that again. There are plenty of opportunities in California. [laugh]
ZIERLER: What was your decision-making as you were rounding out undergrad? Were you thinking equally about grad school and industry?
LYON: No. I applied to grad school, just Stanford and MIT, I believe, and they both accepted me for grad school, but I chose to go to Stanford. I don't think I applied to stay at Caltech, I think, largely because I was fed up with smog. Four years of smog was enough for me. But I worked at the Jet Propulsion Lab as a senior, and I finished all my course requirements a quarter early. I was full-time at JPL during third quarter senior year, and I stayed and worked there all summer. Then I went off to grad school at Stanford in the fall of '74.
ZIERLER: Now, were you thinking at that point that you might go the academic route, or did you see the PhD as the best way to enter industry eventually?
LYON: I'm not sure I gave it that much thought. Pretty early on at Stanford, I decided I really wasn't interested in the PhD. I didn't find Stanford to be nearly as fun a place as Caltech was. It was so big, it felt more compartmentalized. You weren't interacting with a lot of people doing a lot of different things. Everything's too specialized and confined in weird ways, I thought. I think Stanford made a blunder too in how they treated me. When I applied, they asked me if I wanted a research assistantship or a teaching assistantship or just a fellowship to pay my way without doing any work. I told them, in order, I'd prefer a research assistantship and then teaching and then, yeah, if you must, just give me money, that's fine. [laugh] But they just gave me a fellowship. They didn't give me a research assistantship, and so I wasn't really engaged in the stuff that was, to me, the most fun.
I ended up taking classes again. I had an opportunity to take qualifying exams for the PhD, and I just said, "Yeah, no thanks." I just decided to go to work. There's plenty of things going on in Silicon Valley. The coursework there was mostly about information, and communication theory, and digital systems, and things like that. That stuff was all going on in Silicon Valley. Silicon Valley in the early '70s was a military industrial complex place. It had not yet made the change to where it was dominated by more consumer and an industry type products. Most of the electronics was going to the government defense industry. A lot of the signal processing development was about signal analysis for spying on the Russians, and all this kind of stuff was going on. I ended up with three job offers to work in kind of communications and information stuff, two of them in Silicon Valley as part of the military-industrial complex, and the third one was to go to work for Andy Viterbi in San Diego at his company, Linkabit. He later founded Qualcomm there. Linkabit was also part of the kind of military-industrial communications complex. These were three good possibilities. I chose Stanford Telecommunications Incorporated just down the road from where I lived in Silicon Valley. In retrospect, I consider this to be my first big mistake. I should have gone to work for Andy Viterbi—big name, smart guy—but I didn't want to leave Silicon Valley at that point. San Diego might've been great, but I stuck around and have continued to stick around Silicon Valley.
ZIERLER: The road not traveled, have you ever thought about what your career might've looked like had you gone down to San Diego?
LYON: Yeah, I've thought about it. I assume I would've probably gone with Viterbi when he founded Qualcomm, and I would've probably done very well financially. [laugh] Not that I'm hurting here in Silicon Valley, but I'm not, like I said, I'm not among the billionaires. I have lots of billionaire ex-friends and stuff like that [laugh] or billionaire friends and ex-friends, I should say. Someone described it at one time as I never collided with money. That's probably one thing that would've been different. The other is I've had so much fun in working in research labs and inventing things in a variety of fields. If I had gone with Viterbi, there wasn't that kind of variety of weird things to work on down there. I would've been more of a telecom expert probably.
ZIERLER: Tell me about your initial work. What did you work on first?
LYON: At Stanford Telecommunications, the two or three main things I worked on there were a data communication system for—if I remember correctly—the Space Shuttle Ku-band uplink system. I ended up delivering my hardware to Johnson Space Center in Houston, and getting it accepted and qualified and stuff. It was a prototype for stuff that would maybe end up on the Space Shuttle, which wasn't up yet. I don't know what happened after that. I don't know if they used what we did. But we had a contract to develop prototypes for things like that. Then I worked on GPS test transmitter and test receiver. This is before there were any GPS satellites up. Rockwell, I think it was, was making the satellites, and we were making boxes that would pretend to be satellites, test transmitters, and that would pick up, receive signals, and do the kind of processing you'd need to do in a receiver. I designed what was STI's—Stanford Telecommunications, Inc—what was their bestselling product for a number of years, the GPS test transmitter, because they sold two of them. [laugh] Everything else, they only sold one of.
ZIERLER: [laugh] I was waiting for you to say hundred, thousand, million. [laugh] Two, you mean two? [laugh]
LYON: No, they sold two of them.
ZIERLER: [laugh]
LYON: They might have sold more later, I don't know. I burned out there. I was only there a year and a half, and working pretty intently on building this hardware and testing it, designing it.
ZIERLER: What was the funding structure there?
LYON: It was all government contracts, mostly cost plus fixed fee and things like that. The guys that started it had come from Ford Aerospace, where they were used to government contracting. They knew how to play that game and make money at it. After I left some years later, the company, it kept growing pretty good on government contracts, and eventually went public, made a bunch of money. But I didn't get anything off of it because my deal was when I left, I had to sell them back my stock at what I paid for it. [laugh] That was a bum deal as my introduction to Silicon Valley start-ups had sent me a completely wrong signal. [laugh]
Xerox PARC and the VLSI Revolution
ZIERLER: [laugh] Then is it Xerox PARC next for you?
LYON: Right. When I was working for STI, I came to visit Caltech one day, as I liked to do. I went to the lecture hall in Steele, the same room where EE 3 and EE 4 had been taught, a nice lecture hall in Steele. There was a talk by Ivan Sutherland, who they had just hired or were in the process of hiring to run the new Computer Science Department. Carver was there, and Ivan was there, and Ivan's brother, Bert Sutherland was there. Carver introduced me to Bert and said, "You should hire this guy." Bert was managing one of the labs at Xerox PARC. Carver and Ivan and Bert were putting together a collaboration between Caltech and Xerox PARC to do VLSI design stuff. I just happened to pop in at the time to come to Carver's mind to get pointed at Bert, and they said, "Yeah, come on over and interview." I did that, and I interviewed with Lynn Conway, who was heading the VLSI systems group.
ZIERLER: Now, were Lynn and Carver already collaborating at this point?
LYON: This was the start of it. I'm not even certain that they had met at that moment. But it was around that time that the PARC-Caltech deal was being put together, about the time that Ivan came in to head computer science. I think maybe, yeah, probably Carver had already gotten the consulting gig at Xerox, and had been there, and met Lynn, and knew Bert and so on. It was getting started.
ZIERLER: What were some of the big exciting things happening at Xerox PARC when you joined?
LYON: That's a good question. My focus was on what Lynn's team was doing in LSI design. The big exciting things to me were what had already happened there. They already had a personal computer for every researcher, on a local area network, with laser printers, and file servers, as well as time-sharing machines, and email, and connection to the ARPANET. All this stuff was in place. This was in '77. Most of this most exciting stuff about designing their own Alto computers and networks and printers and servers and all this stuff had happened by '77. When I went there, there was a lot of cool stuff going on. Alan Kay's group, the Learning Research Group, doing Smalltalk, was working on their next generation of Smalltalk. A lot of smart guys down the hall in the computer science lab under Bob Taylor were doing all kinds of things. Bert Sutherland's System Sciences Laboratory, which had Lynn Conway's group and Alan Kay's group and some other stuff—I'm a little fuzzy thinking about what was most exciting at that time—oh, and the imaging sciences lab with Chuck Geschke and John Warnock, who later founded Adobe Systems, that was going on.
But most of what was going on there took me a long time to become aware of and to understand the context, because I was not coming from computer science. I was not coming from research, really. I was blown away by what was happening in VLSI design. The concept when they hired me was not to do research in VLSI design or to do research in speech recognition, but rather to work with the people doing that, and design a chip to implement what the speech recognition guys were doing. I was more like hired as an implementation engineer. I learned speech recognition from George White and Raj Reddy, who were some of the early pioneers in that field. George White was there at PARC, and had a couple guys with him. That group fell apart. They all left. George got hired away by another research lab, and another guy, one of his guys kind of left me holding the bag. I learned about speech recognition, and then I got left on my own. My job was still, like, built some chips to show off what VLSI can do, and speech recognition is a good domain for that because it's computationally intense.
Engineering Accessibility
ZIERLER: Dick, let's translate all of that. What does that mean that it was good, it was computationally intense? Why is VLSI good for speech recognition?
LYON: The computers at that time were pretty wimpy, and they couldn't do the kind of processing you needed to recognize speech. They could do it offline. They'd take a minute to recognize a second of speech, but it was a couple of orders of magnitude away from being able to do real-time talking to computers. But with chips, you could design things as parallel and fast and efficient as you like, and that was the point that we were demonstrating in that group. A custom VLSI could solve big problems that the programmable computers of that time could not. When I got left as the speech recognition and VLSI guy, looking at what they were doing, I thought, "Man, this would sure work a lot better if we understood how hearing works." I signed up for a graduate level class at Stanford. They had this program at Stanford where people in Silicon Valley could take courses either on campus or remotely through their television network. It's called the Honors Coop Program, which basically meant you're working for a company, but if a company pays enough money, you can take a class. I did that.
I took a class on biological information processing. As my term project, I wrote a paper on "A Signal Processing Model of Hearing" in 1978. It's an outline of a lot of my career since that time. It's like here's kind of how it works, and here's how we can build it. I set to work designing chips, and went through various generations of architectures to work on that problem, and kind of gradually moved from highly specialized architectures to more programmable architectures. The whole industry ended up moving in that direction, away from highly specialized things, and toward more programmable things. Ivan Sutherland has a perspective on some of that kind of work in the graphics field with what he calls The Wheel of Reincarnation, where people would design specialized accelerators for things like line drawing or shading or whatever, and then in the next generation, the computers were fast enough that you could move those functions into software, and make them a lot more flexible. People kept designing specialized hardware, and then the general purpose stuff would get so much better so fast that it would take over those functions. I think Ivan and I both learned that kind of the hard way that designing specialized processors is not a long-term approach. It always gets bowled over by Moore's law and more general purpose processors. Even today with the AI chips, they're not very specialized. They're very programmable, but they're very parallel.
ZIERLER: What was the consumer end idea for the speech recognition? What devices, what audiences was this looking to reach?
LYON: We pretended we had products in our view, but really we were in a research lab, and we wanted to show off what we could do with VLSI. That was the main motivation there. There was a lot. There was quite a community of people doing speech recognition, and they're all more or less doing the same thing, talking about the products that could come of it. What eventually did come of it was mostly within the telephone network, which was where a lot of the support and a lot of the research was anyway. Gradually, over the years, they phased out operators in favor of more automated systems. Then those migrated into voice response systems for replacing customer service agents with these horrible phone trees and so on. The kind of speech recognition where you can actually talk freely to your computer, and get answers, took a lot more years to get there. But everyone was envisioning it, talking about maybe doing it, but it didn't really happen until the 21st century.
ZIERLER: How long did you stay with the speech recognition research?
LYON: I've been in and out of it. I was at PARC from '77 to '81. When I left, it was to head a speech recognition research group at Schlumberger Palo Alto Research across the street. [laugh] I went from being in a VLSI group to focusing on speech, which was the part of the job that excited me more. Well, yeah, I don't know if that's true. They both excited me a lot. When I went to Schlumberger, I spent a lot of my time doing chip design because, again, it was specialized hardware, or less specialized, but I made a programmable DSP chip that was designed to be good for implementing the auditory models that I wanted for speech recognition, but it was much more generic programmable DSP. I ended up writing some papers and book chapters on that, and making chips that worked, and a SIMD machine with compiler in Lisp and things like that.
ZIERLER: Dick, at Xerox PARC with Lynn's collaboration with Carver underway, did that affect what you were doing? Did that affect in real time how VLSI was developing?
LYON: Absolutely. I was a very close partner in that collaboration. I ended up writing part of a chapter in their book, and implementing a lot of the VLSI design cells and tools that were used to support the multi-project chip effort, and supporting Lynn in teaching her classes, and so on. I was Lynn's kind of right-hand man in that. Carver would come typically once a week for a day, and work with us. It was a time of intense collaboration with them. Really, the VLSI work was great fun, and I came to be known as the VLSI guy. But the speech and hearing stuff still had my love, and it was getting shortchanged in terms of the attention of the organization and so on. That's why I allowed myself to be lured away to Schlumberger to change the focus a little bit.
ZIERLER: What was the collaboration from your vantage point between Lynn and Carver? What did each bring to the table, and then what did you add on top of that?
LYON: The main thing Lynn brought, I think, was her experience with designing design methodologies. She was into systematizing the knowledge, and teaching it in a way that people could easily pick up and use. We didn't know this at the time. I assume you know something about Lynn's background. She had done this when she worked for IBM in supercomputers. She had systematized there the different methods that different designers would use to design logic and circuits, and they were mutually incompatible. She defined some standards for how the logic was going to be defined and implemented so that they could make things that work together correctly. This idea of designing a methodology and sticking to it was really big with her – and simplifying it. She really pushed the simplified design rules. Carver was in on all that, and it was never very clear to me exactly who contributed what part of it, but it was a strong collaboration. Carver brought a lot more experience doing chip design. He knew the physics and the lithography, the manufacturing, the circuits, and a fair amount of experience as well in simplifying and structuring the design methods. They brought a lot of stuff that was similar, in some areas, complimentary, and it was a good team. Lynn did a lot of the grunt work to build the book in soliciting authors for some of the advanced chapters, and doing all the work of putting it together. Carver did some of the writing and a lot of the reviewing, of course.
ZIERLER: Did you see their collaboration, the motivations, was it primarily educational, or were they doing original research as part of the collaboration?
LYON: It's really both. I think that primarily educational is probably fair, but research was needed to take what's known about VLSI, and convert it to stuff that can be taught, systematize it. The research is in systematizing methods and practices and engineering into a set of ways of doing things that would reliably work; robust, teachable, workable approaches to design. I don't know how to characterize it. In some ways, the research was sociological. It's like how do you build a community around this stuff? It really needs to take off, and be taken out of the hands of the semiconductor companies where there's just not enough designers or design capability compared to all the students that we could get through universities. There was not a lot of social move in that direction. Carver had been teaching VLSI design at Caltech, but it was still a very tiny community, and his students would go work at Intel or something. It wasn't expanding the scope of what schools and companies could do VLSI design. Lynn's vision was to go big with that. One of her phrases too is go meta. Whatever it is you're doing, let's figure out what the meta thing is, and push that. She made a lot of amazing stuff happen there.
Creating the Optical Mouse
ZIERLER: When does the optical mouse enter the story for you?
LYON: That all happened in 1980. One of the other university people that was working with Lynn's group was Carlo Séquin from Berkeley. He had previously been at Bell Labs where he worked on charge-coupled devices for image sensing. He had written a book on Charge Transfer Devices. He knew a lot, like Carver did, about the kind of physics and manufacturing of circuits. He wasn't as focused on strict digital methodologies, like Lynn was. I learned a lot of kind of analog stuff and tricks and things from Carlo. He taught me how light interacts with silicon, and how you would make a photo sensor, and a few things like that. There were some guys at PARC who were fed up with their ball mice that would get dirty, and stick, and have problems all the time. Several different guys had already filed invention proposals on how to make an optical mouse. I looked at those. I was able to get hold of them, and read them, and talk to the inventors, and say, "Does this really work?" They said, "Yeah, no, not really."
ZIERLER: [laugh]
LYON: [laugh] Basically they were half-baked ideas, but they got me thinking. Then I thought, well, we could do this all on a chip. That would make it cheap. We could do considerably more complicated stuff than what they were doing in these simple-minded invention proposals. Carlo showed me how to detect light on a chip. Those ideas gestated in my brain for quite a while, and I think kind of one night at home when it clicked what the key step was that I needed to put all that together through what I called lateral inhibition circuits. I ended up putting in the hours to design the whole thing, did the layout. ICARUS, the IC design layout tool that we had on the Alto, was designed and implemented by Doug Fairbairn with Jim Rowson, who was a Caltech grad student under Ivan and Carver. We had this nice tool, well, as nice as you could get on a black-and-white bitmapped Alto screen anyway. It's awkward. But I did the layout, made gigundous check plots on the Versatec printer so we could see if I did it right, and looked for mistakes. I managed to get Chuck Thacker from the Computer Science Lab to pore over my circuits and layout. He spent hours on it, to see if I implemented what I said I was implementing, and see if I made any mistakes. He found two little mistakes that would've killed the thing, basically, so I owe him a great debt for finding those.
ZIERLER: What were the mistakes?
LYON: Layout errors. I had these programmable logic array structures where you put the bits – it's like putting bits in a read-only memory. You put these little rectangles of what we called green, which was a certain stipple pattern on the black and white display, a little green area that would overlap a red area that would make a transistor. You're putting in like transistors for ones, and no transistors for zeros. It was all done with geometry. If the ones and zeros were right, it was going to work. But I had a pair of green flashes transposed from where they were supposed to be, and Chuck found it. For someone to put in the effort to check things at that level was pretty astounding. At the same time, we had – through Lynn teaching VLSI at MIT – we had students developing tools like—what do they call it?—a schematic checking tool, layout versus schematic, or something like that, tools that could find these kind of errors automatically. But we didn't have that running yet internally. We got it done, and it worked. When the chip came back from manufacturing, I got packaged chips, and I said, "Now go test them." I realized, in my zeal for making an optical mouse chip, I had forgotten that I would have to test it. The chip had no inputs, so I couldn't put it on a tester and give it some inputs and see if it gives me the right output. The only way to test it was project a light pattern on it [laugh], so I did that. I set up a little lens over the chip, and I plugged the chip into the mouse port on my Alto because it was designed to be directly compatible, and I projected a little pattern onto it, and the cursor on the screen moved. I said, "OK, it works." [laugh]
ZIERLER: That's it?
LYON: Yeah, basically. Then the subsequent version, I worked with Martin Haeberli, and he added a testability feature, so subsequent versions we could put on a chip tester. [laugh] We just added a little transistor for each photodiode that would simulate light hitting it, so you could pretend you were projecting a pattern onto it, and test it that way.
ZIERLER: What was the timeline for optical mouses to completely overtake the ball mouses? I remember as a kid, I used the ball mouses. The optical mouses didn't really come till later.
LYON: Yeah, it's crazy. There's almost a 20-year lag from when I invented it until optical mice became dominant. The reason is that Xerox never let it go anywhere. Xerox shipped products with the optical mouse, their office systems and high-end copier-duplicators, and machines that ran Lisp, and they sold some through Tektronix on Smalltalk machines. There were optical mice out there on a number of products, but none of these were high-volume products. This was all before the PC, or coincident with the PC, but before the Macintosh. The Mac came out with a ball mouse, and the whole PC industry pretty much used mechanical ball mice until the optical mouse got reinvented at Agilent in like 1999. I think Microsoft first had a—what did they call it—Microsoft Smart Mouse or something, and then it took off.
ZIERLER: Why did Xerox PARC sit on it for so long?
LYON: [laugh] Why did they do all the things they did?
ZIERLER: [laugh] That's fair. [laugh]
LYON: Just fumbling the future is what they're known for.
ZIERLER: But wasn't it understood objectively to be a superior technology to the ball mouse?
LYON: Yeah, by people who knew. Everyone in Xerox and all the Xerox products used it. It was much less expensive than their ball mice, and it didn't gum up with dirt, and it was a good thing. Now, it required a pad, a patterned paper. Later, when they reinvented it later with quite a few generations of Moore's law doubling in between, you could put many more photodiodes on a chip. My optical mouse had a four-by-four photodiode array, 16 photodiodes. When they redid it, reinvented it, it was a much more complex chip with, I think, thousands of photodiodes, and much more elaborate processing. They did it what I'd call the hard way. But as a benefit, they got to operate on surfaces with any subtle texture. It didn't require the pattern that the original optical mouse required. I don't think the requirement of a pad was such a stumbling block as to explain why others didn't adopt it, but they didn't. Xerox had patents on it, and they weren't licensing it. They weren't trying. They weren't pushing it. They could have sold them.
ZIERLER: In 1999 with Agilent, as you were explaining, did they take credit for this, or was there some recognition of your role, of Xerox PARC's role?
LYON: As far as I know, they were not even aware of it. There were some patent suits over the years in which my patents were brought up as prior art to show that some of what Agilent was claiming was not valid. They didn't cite my patent, didn't seem to be aware of it.
ZIERLER: Now, was this work sort of a side project to the speech recognition? Did you see some integration there?
LYON: Yeah. A lot of what I did there and what I've done over my career, I guess, is fun side projects, like, get an idea and you pursue it. [laugh] It was an idea that clearly had support where I was. Lynn said, "If you can do that, that's going to be cool. [laugh] People will like that," so I did that. I had some other side projects too, like, people were working on developing the 10-megabit Ethernet. We had the pre-standardization 3-megabit Ethernet running at PARC, and they're working with a consortium with Intel and DEC developing the 10-megabit Ethernet, which required new chip designs and so on. The companies were designing these two-chip sets with a bipolar integrated circuit for driving the wire, and picking up the signal from the wire, and an MOS integrated circuit for doing all the processing computing stuff. I thought that was dumb. I thought we could do this all on a MOS chip. I'll show them how. [laugh] I made some circuits to show them how, and got some patents out of that as well.
ZIERLER: Why ultimately did you decide to leave Xerox PARC?
LYON: It was really to lead a group doing speech recognition. It was a promotion and a raise and refocus on stuff that I felt was my own research area.
ZIERLER: Did the move actually help? Were you able to do what you had wanted to achieve?
LYON: Yeah. It was it was a fantastic lab for quite a few years. A lot of good people went through there. It didn't last. By 1988, it had been through a succession of managers, kind of fell apart, and Schlumberger decided to move what was left to their computer science research lab in Austin. I don't know. We had like 50 people or something, and only 6 of them moved to Austin. The others dispersed within Silicon Valley. Thirteen of us ended up at Apple. [laugh]
ZIERLER: Was this really the origins of Austin becoming a tech center?
LYON: No, I think they already—like I say, they only moved six people there. But Schlumberger already had a research lab in Texas. There was stuff going on in Austin for a long time, and Dallas of course with Texas Instruments.
ZIERLER: Could you keep up the VLSI collaboration with Carver and Lynn from Schlumberger?
LYON: Yeah, sort of. With Lynn, I got involved in her—she went off into more kind of AI and Lisp. She had a little challenge called the LOOPS Challenge, where people would try to write little artificially intelligent programs to move vehicles around in a game scenario or something like that. I ended up winning one of her competitions on that. With Carver, it was the beginning of a closer relationship because he invited me to come down to Caltech one day a week. I started doing that in '83, and continued that as a visiting associate for about 13 years. I was basically spending a whole day a week with Carver for a long time. That was fun.
ZIERLER: What was he working on in that period?
LYON: At the start, he was in computer science, but he was putting together this new group called computation and neural systems, CNS. I was a member of the CS faculty for a while, and a member of the CNS faculty for a while. It was a great position. Visiting associates have no duties, and no pay, and no rank.
ZIERLER: [laugh] Just the fun stuff.
LYON: They would pay my expenses to come to down there once a week, so that was cool. My employers – I did this both with Schlumberger and with Apple – I just basically had to send them a letter to say, "This is what I propose to do. Is that OK?" They'd say, "Yeah, sure."
ZIERLER: Dick, did you see the biology interface, the neural networks, was that particularly relevant for speech recognition technology?
LYON: Yes and no. The speech field has been complicated because, over the years, they've slowly made some changes that make the front-end signal analysis more like the auditory system. But they've been really sluggish in that, and they've been reluctant to go further. They've been stuck, I think. for a long time. Even at Google, I've had a lot of difficulties in getting relevant improvements into some of our systems. I have gotten some in, but it's been a chore. The field has mostly decided that they don't care that much about front-end processing anymore, which is where my interests are, and it's all about the machine learning back-end stuff, and not that many people that want to mess with the front end. But there were some times when we had to. I did some things that saved our bacon.
From Apple to Foveon
ZIERLER: Tell me about moving over to Apple in 1988.
LYON: I had interviewed with Apple twice when Steve Jobs was there, before they kicked him out, and didn't really want to work with him. He was kind of a jerk. [laugh] But he was gone by '88. Shane Robison had been one of our researchers at the Schlumberger lab. I was leading the speech recognition team, and Al Davis was heading a computer architecture team that Shane was a member of. They tried to get Al to be the manager for the lab, and he said, "No, I don't want to do that. Maybe Dick will do it." They asked me if I would do it. I said, "No, there's no way I want to be the manager [laugh] of this lab." I talked to Al about it, and we jointly said, "This guy Shane seems ambitious. Maybe he could be our manager." We promoted him over our head to be our manager, basically. That was the start of his meteoric rise I guess. He went to Apple, and headed up a lab, and hired me to come over and join him. He was in charge of the Newton project for a while, and he left there, he ended up being the CTO at Compaq, and arranged the sale of Compaq to Hewlett-Packard, and then he was CTO of Hewlett-Packard, and CEO of some other company. He did real well, but Al Davis and I launched him by promoting him to be our boss at Schlumberger.
ZIERLER: What were your motivations in moving over to Apple? What could you accomplish there?
LYON: They had a research lab that was interested in speech, so I could keep working on speech. I could work with people like Shane and others that I had known at Schlumberger. It was a fantastic place. It was kind of a roller coaster. But, at that time, John Sculley was a hero, and the Macintosh was doing well. I went over, and the first week I was there, they had a big company meeting at Flint Center in Cupertino. John Sculley stood up in front of the employees, and talked about how the business was going, and got a big standing ovation and everything. It was a very exciting time for Apple. Then he talked about his plan for hiring. He says, "We're going to hire 3,000 people this year." I said, "D'oh, that's not going to work. You can't integrate that many people in a company this size." He said, "I figured out the rule for how to make money in a high-tech Silicon Valley company, which is to keep the gross margins up at 55%." I said, "D'oh, that's not going to work. You've got to go for market share."
On the one hand, things were going well, and it was exciting when I came in. On the other hand, I could see this company was not managed in a way that was healthy. It wasn't but a couple months later that they had a hiring freeze. So much for hiring 3,000 people this year. I was there for nine years, and it was quite a roller coaster. I survived a number of layoffs, and fortunes of the company going up and down, and CEOs being kicked out and replaced by other CEOs – Spindler and Amelio. Then they were dying. By '96, they were dying pretty bad. Steve Jobs engineered this thing where they would buy – who'd they buy? – Next, I guess, and take in some of his technology. He engineered his takeover, and kicked out Amelio, and so on. But in the process of doing that, he laid off all of research, so that was the end of my time at Apple.
ZIERLER: What year was that?
LYON: '97.
ZIERLER: Now, during this time, you're still collaborating with Carver. This is when you get into the cochlear research with him?
LYON: That was actually quite a bit earlier. By '97, Carver was on his last grad students. His final grad students were all finishing in '97, a couple months later after I got laid off. I was still going there and working with him and his grad students, and I continued doing so after being laid off. That's when we founded Foveon, that summer of '97. Carver and I and some guys he was working with at National and Synaptics founded Foveon.
ZIERLER: Was this only in response to getting laid off, or were you thinking about this as you were seeing the ship sinking at Apple?
LYON: No, I was not thinking about Foveon during that time. I didn't know what I was going to do next. I figured I'd find something fun. When I wasn't working for Apple anymore, but I was still working with Carver, at some point, he was behind the scenes working with Synaptics and National, where he had a project going on image sensors. He engineered a deal to get people and IP out of those two companies into a spin-out, and invited me to join them as their chief scientist. It just kind happened that way.
ZIERLER: Let's start with the name. Why Foveon? What does that mean, Foveon?
LYON: To start with, it was Foveonics, and it means like electronics modeled on the fovea in the retina of the eye. It was a year or two later that we thought Foveon sounded much more succinct than Foveonics, and we changed it. When he invited me, I'll always remember what he said. He says, "Dick, this isn't going to be a cushy job like you're used to in those research labs. You're going to have to work."
ZIERLER: [laugh]
LYON: I said, "Yeah, I know that," so I did. It was a lot of work.
ZIERLER: What was the game plan? What was the mission for the company?
LYON: Basically design, build, sell cameras that were digital cameras good enough to be used in professional portrait studios. There wasn't such a thing at the time. We did that. We sold our first portrait studio camera, I think, at the end of 1999 to a portrait studio in my hometown, Sheldon of Los Altos on Main Street, the same studio that had shot portraits of my little kids some years earlier. It was an amazing contraption. It used a color separation prism that I designed, and it used some three image sensors that Dick Merrill designed, and it had a bunch of software that I did with the software team. It was built around a Dell laptop computer. It's this amazing contraption where the computer provided the control and storage and viewfinder and interfaces to run the camera. It was this big contraption that sat on top of a tripod; not the kind of thing you'd walk around with but it was perfect for a portrait studio. I went by Sheldon of Los Altos a few years later, and I talked to the photographer, Eddie, and he was so happy. He said, "Dick, we've made so much money off this thing."
ZIERLER: Wow.
LYON: I said, "Really, why?"
ZIERLER: Why? Just because people love the photographs?
LYON: No. [laugh] It was because people can see the photographs when they're in there for their portrait session. In the old days, they shot on film, and then they'd say, "Come back and look at proofs in a couple weeks." Being able to complete the sale while people are still in there with the whole family, all dressed up, and excited about having their picture taken was why he made a lot of money. It didn't really occur to us that that was the relevant dynamic, but that was huge. But we never sold very many of those. In about 2000, we switched over to a completely different technology called Foveon X3, which was a scheme in which we layered the photodiodes in the silicon to detect red, green, and blue all at the same location, rather than either the color separation prism that we used in the other camera or the mosaic of filters that other people were using in their digital cameras. This was a way to get the high-quality, alias-free color imaging on a single chip. This connects to a story that I got from John Pierce. Pierce was, to some extent, largely known for his development of the traveling-wave tube amplifier. He would say sometimes—and I think he said this in print maybe in his New Yorker profile in the '60s—he said, "Rudy Kompfner invented the traveling-wave tube, and I discovered it."
ZIERLER: [laugh]
LYON: At Foveon, Dick Merrill invented the X3 technology, and I discovered it. As my job in managing the IP portfolio, I had inherited a stack of invention proposals from National Semiconductor where he used to work, because they gave us the people and the technology and the IP for the image sensor project. Part of my job over time was to go through this stack of patents in progress, and patents half-written, and invention proposals not yet analyzed, and so on. I found this one about how to sense red, green, and blue all in one place in the silicon using wavelength-dependent absorption of light and silicon. I took it to Dick Merrill, and I asked him, "Did you guys try this?" He says, "No, we never built it." I took this to Carver, and I said, "Do you see this? Do you think this could work?" Carver says, "Oh, Toby [Delbruck] and I tried something like that one time. It didn't work." I went back and looked at what he and Toby had done, and I analyzed it. I came back to him, and said, "No, look, this is what you and Toby did. This is what this one does. It's different." He says, "Oh, I see. That could work?" I said, "Yeah, this could work." We set about making it. The first time we made an image sensor with it, we came back. My job is generally take the lousy signals we're getting off the silicon, and see if I can make a pretty picture out of them, and so I tried that. Dick Merrill built the chip, built the camera, got me an image, and I played with it. I finally had to give up. I told him, "No, it's not going to work, and here's why." Merrill says, "How about if we do it this way?" He makes me another sketch. I said, "If you can do that, that will work. I know it will." He says, "I think we can do this. Let me talk to my guys in the fab." He goes back to his friends at National Semiconductors, and says, "Can you guys make this?" They said, "Yeah, let's try it." He designed another chip, got the chip back, built a camera around it. I sat in front of it. He took a picture of me with a color chart and some flowers and things. I went to process the data, and it's a beautiful portrait, and I still use it. [laugh] The first test chip worked flawlessly.
Even before we had that, we had sold the idea to Sigma, we were so sure it was going to work. They were already committed to building a camera around it. This was super exciting times. When I interviewed at Google, one of the interviewers asked me about my career. He asked, "Have you mostly worked on developing your own ideas or the ideas of others?" I had to think about that for a while, and I realized it's probably more developing the ideas of others, some of each, but like the Foveon X3 was a good example of that. Foveon in these early years of Foveon was the most fun and exciting part of my career, by far. Carver, I think, was a big part of the reason for that, because working with him was always exciting, and the technology we were working with there was so different from other stuff I had done, working on learning about optics, and colorimetry, and physics of light absorption in silicon, and all this stuff that had to come together and make beautiful pictures that would make customers happy. But it all worked out. It was a ton of work and a ton of fun.
ZIERLER: Now, did you stay at all with speech recognition research during the Foveon years?
LYON: Not at all. People in the field wondered how I dropped off the map. When I reappeared and went to Google in 2006, and went to the Interspeech meeting, people were surprised to see me. It's like, "Where have you been for the last decade?" [laugh]
ZIERLER: Did you keep up with the literature at all during that time?
LYON: No, not really. No, I was busy. [laugh]
ZIERLER: How did Foveon fare in the long term? Was it a successful company?
LYON: They had some successful products but, no, they were not a successful company. They never sold enough to make a profit. In the end, they sold the company to Sigma, which was their only customer at the time. We were making the image sensors for the Sigma line of cameras. They've put out, I'd say, more than a dozen cameras with Foveon sensors. When I left in '06, I was in charge of the image sensor design for their next generation camera, and that came out good. I can't even remember what the name of that generation camera was. But they did several more since then, went through some management changes, lots of ups and downs. But, in the end, they sold it for a fire sale to Sigma.
ZIERLER: Now, in the middle of all this, of course, is the dot-com boom and then bust around the turn of the century. Did that register with you? Did that affect what you were able to do?
LYON: Yeah, it did. It made it a lot easier to get to work in the morning when there's no traffic in Silicon Valley.
ZIERLER: [laugh]
LYON: Things were tight. But around '99, 2000, 2001, this was when we were developing and delivering the first prism camera, and doing the research on the Foveon X3. It made it hard to get money, and it made it hard to get sales, so it was a tough time. But we had enough money in the bank to get through it.
ZIERLER: What do you see as the key achievements for you and for Foveon?
LYON: In the long run, it's not clear anything we did matters, because the technology we developed for Sigma, while it worked well and made its end users very happy, it didn't cause a big shift. It didn't create a big enough market for Sigma. It didn't change the industry. It didn't change the way photography development went forward.
ZIERLER: We don't see an afterlife of the X3 sensor?
LYON: Not much. They tell me it's not dead yet. They shut down the Foveon operation in Silicon Valley, and moved it back to Japan where Sigma is. They claim to still be working on the next generation, but it's been a few years, and I'm not convinced it's alive. It might be alive, but it's not going to be a big deal. To Sigma, it was a big deal because they wanted to get into the digital camera business, and they needed a hook. They didn't want to just be an also-ran with the same kind of sensors as everyone else. You can't compete with Canon and Nikon and Sony that way. They couldn't compete with us either, even though they had some users that loved it and swore by it, and they still do. It's just not enough to make it a commercial success. I was proud of what we did. But, in the long run, I think it doesn't matter. The prism-based camera we did before that, another dead end. Oh, by the way, I got another call from Eddie at Sheldon of Los Altos last year about this time. He says, "I got this old Foveon camera. It's been in storage for 14 years." He used it for a good run of about nine years before the Canon cameras got to be good enough to take over the business. He said, "You know anyone that wants it?" I checked with my old Foveon colleagues. Responses like, "The wife will kill me if I bring home more junk."
ZIERLER: [laugh]
LYON: I said, "Maybe I'll take it." I said, "Are you looking to sell it or give it away?" He says, "I was hoping to sell it." [laugh] I said, "Will you take $150 for it?" He says, "I wanted 200, but since it's you, Dick, you can have it." I took it, and I fiddled with it. He gave it to me with no cables, no tripod, no lens. It was just like OK. I fiddled with it, got some parts, put it together, made it work. It's still working. I asked the Computer History Museum if they would like such a thing in their collection, and they said yes. I delivered it to them along with all kinds of other Foveon bits and pieces and memorabilia and software and archives, and so I got it out of my garage. [laugh]
ZIERLER: Now, did the timing work out in terms of the collapse of Foveon, and your desire to get back into speech recognition? Did that sort of align well for you?
LYON: More or less, yeah. I had some friends at Yahoo! that were looking to hire me. I was looking to get out of Foveon, and my wife said, "If you're thinking about going to Yahoo!, why don't you talk to Google?" I said, "Google? They do research?"
ZIERLER: [laugh]
LYON: She said, "I don't know. Why don't you call Eric, and see?" I emailed Eric Schmidt.
ZIERLER: This is Eric Schmidt?
LYON: Yeah. I emailed him. He called me right back, and said, yeah, I should come in and interview. He set up an interview with Alan Eustace, his VP of engineering, and Alan set up interviews with other people. I figured I'd have a good in with Eric because he was on record saying to one of my brothers that if he ever had another chance to hire another Lyon brother, he would. He worked with Bob and Tom at Sun Microsystems. My brother Bill told me this, that he had said it to him. Eric knew who I was, so he was willing to give me a shot.
The Origins of Machine Hearing
ZIERLER: Even that early, Google was involved in research? They weren't just exclusively focused on search?
LYON: Yeah, they had research. I think Wayne Rosing used to head up the research here. He was an old guy from Apple. I didn't really know him. But he was not even here anymore at that point. They had brought in Peter Norvig. They had research. At that time when I joined, there were 70 people in research, and they all reported to Peter Norvig. It was a kind of small, flat organization. That's what I jumped into back in 2006. It wasn't specifically to get back into speech recognition. But, at that point, I thought there's all this AI neural net stuff that I had done a bit of at Apple on handwriting recognition, and a lot of media on the web that needed to be indexed somehow. I figured some kind of image analysis and/or sound analysis was going to be useful, and I should work on that. I spent some time in my early months trying to figure out who knew what. I started a little mailing list called Signal Understanding to try to organize people that were working in this space. What I had discovered pretty quickly is there were quite a few people already that were really good in looking at image analysis and video analysis and stuff like that, and nobody working on sound. I said, I guess that's my area. But, in truth, I had a really hard time getting started at Google, not really being a software engineer. Even though I had learned to write C++ code at Foveon, I was out of my element. I didn't know anything about internet programming and all the things that people here did.
I was struggling for exactly what to do, when I was walking down the hall one day, and through a window in one of the conference rooms, I saw one of the guys that I had interviewed with. I thought, I'll just let myself into his meeting, and see what's going on. I went and sat in the back of the room, and they're talking about how they took some cameras from the book scanning project, and strapped them on top of a van, and drove around town taking pictures. They were looking at these pictures, and they were awful, and they couldn't figure out what was wrong with them. Toward the end of the meeting, I raised my hand, and I said, "I'll come back next week, and make you a presentation to explain all this," so I did that. That was my introduction to the Street View project, which was called City Block at that time. A couple years later, they made me manager of the camera design team. That's how I got into doing camera design at Google. I thought I was getting away from that when I left Foveon, but I got back into it because there was an obvious need for what I knew. At the same time, I was getting some traction and getting some work going with hearing models and sound analysis. I was running back and forth, managing a couple of guys in research and a good-sized team in the Geo organization, doing cameras and hearing for a couple of years. Then once the cameras were under control, I gave that up in 2010, and started writing my book on Human and Machine Hearing, and refocused on that. That's where I've been since.
ZIERLER: The book was part of your overall duties? Did you take a sabbatical for that?
LYON: No. I talked to Peter Norvig, who had written a book on artificial intelligence. I asked him, "Did you have to take time off to do this? It's OK if you write a book on Google time?" He has taken time off to do some revisions of his books, but he basically said it's OK. I said, "What about royalties? Do I have to pay those to Google?" He says, "It doesn't matter. It's not worth worrying about." [laugh] I did it on work time. I've got a copy of the book here.
ZIERLER: Dick, what additional—
LYON: I know this won't be on the audio transcript, but there's what the book looks like.
ZIERLER: There it is, Human and Machine Learning: Extracting Meaning from Sound.
LYON: No, Human and Machine Hearing.
ZIERLER: Hearing? I'm sorry. Yeah, I say learning so much, machine learning, right.
LYON: When I gave a talk at the Caltech EE Department Centennial in 2010, and I was introduced by P. P. Vaidyanathan, he said I was in charge of machine learning at Google.
ZIERLER: Yes, that's right. [laugh]
LYON: It's very wrong.
ZIERLER: It's just the phrase is tattooed in our minds, for better or worse.
LYON: There was a review written of this book, and then the review was republished in another magazine with the title changed to Human and Machine Learning. People are really stuck on that term, which really annoys me, as you might imagine. [laugh]
ZIERLER: Obviously, there are big differences. But I wonder if you could just define what is machine hearing?
LYON: The subtitle here sort of says what it is. It's extracting meaning from sound. If you have a machine that takes a soundwave in, and gets something meaningful out of it, then that's what I call machine hearing. That can include speech recognition and sound event detection and all kinds of things like that.
ZIERLER: Dick, what kind of research did you need to do for this book that might have been outside your wheelhouse?
LYON: It's what I consider to be my wheelhouse. It's what I've been doing since the Xerox days. There's a lot of stuff I learned at Xerox and Schlumberger that went into this. In the hearing field, I see a lot of confusion about how hearing works, so I was trying to tell the straight story, and avoid some of the misconceptions and traps that you see so often in books and papers and chapters on hearing, where they basically promulgate a lot of mistruths and misinformation. I'm trying to set that straight without being too critical.
ZIERLER: What are some of those misinformation themes that keep on popping up?
LYON: The main theme is the belief in the old Helmholtz model. Hermann von Helmholtz was a famous and respected scientist of the 19th century, who basically said that the ear works by having a bunch of resonators that respond to different frequencies, and that the phase of the resonance is irrelevant. We just look at what the ear sends to the brain. How much energy is at each frequency? It's a power spectrum kind of representation. There's a lot of different ways that this could be mathematized and implemented and so on. But this kind of a phase-blind frequency-domain view of how hearing works has caused a lot of harm over the years. There's a famous hearing scientist named Georg von Békésy, who I quoted from his 1974 paper. I've been using this quote in my talks since I was at Xerox in the late '70s. It's in one of the chapters here, and I might get it wrong. But it's something like he said, "Dehydrated cats and the application of Fourier analysis to problems in hearing become more and more a handicap for progress in hearing research."
ZIERLER: [laugh]
LYON: I just love the way he lumped Fourier analysis in there with dead cats.
ZIERLER: [laugh]
LYON: Anyway, a long story there, but there's other misconceptions too. But the constructive part of the book is to lay out kind of a schema for how to build machine hearing systems, kind of a layered architecture, and details of how to build each layer. The bottom layer is a model of sound processing in the human cochlea. There's a suite of open source software that we put out about the same time that does that. The middle part of the book is the detailed documentation of that software, effectively. It's not written like a software documentation, but it's the thought and development behind it, the details of how it works, and so on.
ZIERLER: Dick, what was the readership you were looking to reach? Is this like a textbook? Is this for more a popular audience?
LYON: Not a popular audience, no. It's a very specialized audience of people that actually want to build machine hearing systems, or people that want to better understand human hearing so they can build better hearing aids or something like that. The hearing research field is the audience, and it's a very diverse field because people come at it from very different directions, and work on different aspects of hearing. It's hard to target everyone that broadly, but I never really narrowed down on who the audience should be. I put a lot of stuff in the book to try to bring the different viewpoints together, what I call connection boxes, where I have a little aside that talks about people from the EE side talk about it this way, and from the physics side talk about it this way, and connections to math and statistics and psychology and mechanical engineering and stuff like that. I'm trying to put the stuff in that would help the people from the very disparate backgrounds understand each other better. It's an attempt to teach a lot of important stuff to the field. I'm not sure it's a successful attempt, as I've only sold a little over 1,000 copies, but it's making a dent.
ZIERLER: Have you been able to track or do you get feedback on where your ideas get translated into all of these products broadly conceived in the hearing machine research community? Do we see, for example, are hearing aids getting better? Are there new computer sound interfaces that happened as a result?
LYON: It's been slow. The thing I'm working on now with a collaboration in Australia is to make better hearing aids, and to make it possible to adjust hearing aids better, using better models of normal hearing and impaired hearing. That project uses my technology at the core of it, and it involves Google and five outside partner organizations in Sydney. That's one of my current hot things.
ZIERLER: For medical devices, have you learned or have you gotten at all involved in the regulatory framework?
LYON: Oh, I avoid that but, yeah, I've looked at some of it. I've tried to read the regulations on the new over-the-counter hearing aids, which are by no means deregulated. They're over the counter, but they're [laugh] highly regulated, and it's a mess actually. But that's not my department. I'll do the research side, and if people want to make products, I'll have to deal with the regulatory stuff.
ZIERLER: Dick, what about at Google? Are there Google products for which this research is specifically responsive or relevant?
LYON: If there was, I wouldn't be able to tell you that. Let's leave it at that.
ZIERLER: Okay, fair enough. What about on the camera side, thinking about the ways that Google has advanced or has used cameras for the various products that it's involved in?
LYON: We have a really strong team doing computational photography, and they've done some amazing things with cameras that are orders of magnitude beyond some feeble steps that I took at Foveon to do some of these things, like high-dynamic range rendering, and multiple image combination, and stuff. I sit near them, and follow what they're doing, but I'm not that involved. Early on at Google, I taught a course series on photographic technology, and you can find most of the most of the videos online. It's a series called PhotoTechEDU. The EDU was what Google used for their educational courses internally, so that's where that came from. I gave, I think, the first half dozen lectures myself, and then I had guest lectures for the rest of it. That stuff's kind of long in the tooth now, I guess, but this was a 2007 view of photographic technology. Everything is so much more computational nowadays. But for people that are learning, I think it's still a useful resource. I did it because when I started working with the Street View guys, they didn't know anything about—they were software engineers, hardware engineers, mechanical engineers. They had no optics, photography, image processing, and colorimetry background, so I really did it to educate the people on the project as well as others at Google, and had pretty good attendance. I think I had a pretty good influence that way. It's all out there on the web but, like I say, it's 15 years old.
ZIERLER: Dick, bringing our conversation closer to the present, when COVID hit, working remotely, what was that like for your productivity?
LYON: It was a rough and complicated time for me because before COVID hit, I had my house gutted for a major remodel, and moved into an apartment for a while, a short term, I thought. [laugh] Then with COVID, the work on the house shut down. I couldn't get permits, and inspections, and workers. My personal life was not in great shape, and my work from home life was not in great shape because I didn't have my home. I was in a little apartment. That was awkward, but we managed it. Since that time, it's been up and down with fits and starts of working on the house. It's mostly done now but still has some stuff going on. [laugh] But we've been living in it. In the meantime, we did a lot of traveling during COVID shutdown. We did a lot of times when the house was having noisy things going on. We did a lot of time trying out all the hotels up and down El Camino Real, and trips to Santa Fe, and trips to Maui, and trips to El Paso and Bellingham to visit the relatives, and stuff like that. It was nice to be able to work from anywhere, but work from home was not right for me. But the team stayed – they stayed productive. I took being their manager seriously during COVID, and I stayed in close touch with everybody to make sure everyone felt supported. Then at some point, I realized that doing all these online virtual meetings where everyone was in their own little box actually worked out much better than the style of meetings we used to have where half the team is in a big conference room, and the other half were in little boxes. These kind of hybrid virtual meetings were a mess, and the people that weren't in the room felt greatly disadvantaged, and so on. I thought I should take advantage of this, everyone virtual, to run a course. I ran a course on my book where we did a chapter every other week, a discussion course, a discussion in the sense that I didn't present the material. I got the students to volunteer to present each chapter, and then I was there to help clarify and answer questions and so on. I ran two sessions, one Europe-friendly and one Asia-friendly. I had people from all over the world in a fully virtual distributed class, and that worked out really well. That was really the highlight of my 2021. It basically ran most of the year. I got to be friends with a bunch of people that I'd never met in person all over the world, and so that was a kick. There was ups and downs with the whole COVID thing. [laugh]
ZIERLER: Bringing the story right to the present, what are you currently working on—that you can tell me, I should say? [laugh]
LYON: About a year ago, I gave up being a manager, and went part-time, and worked on putting together the deal in Australia for the multi-party collaboration. I spent five months there from December to May, helping them kick that off. I'm still working with them but remotely from back home, hoping to go again before too long. That's what I see as my main project. Then I have some other things I'm trying to do, connecting with some of the other projects here in Mountain View that are doing sound analysis, and applying some of the same techniques that are motivated by the hearing models, though it's all about hearing these days.
The Legacy of Caltech Creativity
ZIERLER: Dick, for the last part of our talk, now that we've worked right up to the present, I'd like to ask a few sort of broadly retrospective questions about your career. Then we'll end looking to the future. First, let me just start with Caltech, and especially Carver Mead. What has stayed with you from your undergraduate days from the Caltech way of approaching a problem, of doing science that's just always been part of what you've been able to do?
LYON: I think what I learned at Caltech is that most science and engineering problems can be worked by figuring out the right mathematical descriptions, and then working the math. It's an analytical approach. That's part of it. The flip side of that is being able to create new things, which really comes from tinkering and projects and stuff like that, where there's no roadmap. There may be some methodologies and techniques that you incorporate—there have to be—in order to make it possible, and the idea that complicated systems need to be built on stable subsystems, which is something that we learned from biology as well. Take the stuff you know and understand. Invent new stuff that uses it. That's something I did in coursework and projects at Caltech and since then.
ZIERLER: What about accessibility? What have you made possible for people who have various disabilities that can't use the things that abled people can use?
LYON: The main things we've done at Google are two Android accessibility features –apps, if you will. One is basically a sort of a hearing aid, like audio processing in the phone. It's called Android Sound Amplifier. It's not just an amplifier, but it's a multiband compressor, which is like the technology in hearing aids. It amplifies weaker sounds, and not so much amplifies louder sounds. This is what people with normal, mild to moderate neurosensory hearing loss, it's what they need. If you use your phone as a microphone, and you put headphones on, and you're hard of hearing, you can use that like for listening to your TV, or having people talk to you, or whatever. For people in countries where audiologists are rare, and hearing aids are too expensive, this is a big deal, so it actually gets some traction. But the other one we did gets even a lot more traction, and that's called Live Transcribe. That one uses Google's speech recognition technology in the back end. It's basically just an interface that takes what comes in the phone mic, and transcribes the text, and prints it out big. If you're deaf or severely hearing impaired, and you can't hear people talking to you, you just look at the screen, and see what they're saying.
This has changed lives, and it has more regular users than all other Android accessibility features put together. It's huge, and we're still working on it. It was really developed and motivated by a fellow who joined my team, I brought into my team a few years back—five years ago? I don't know—who has been deaf since he was a young child, and he gets by with lip reading. He's not a signer. He doesn't have cochlear implants or hearing aids or anything. But he doesn't get by very well with lip reading unless you're speaking Russian, which was his first language. His second language, English, and his third language, Hebrew, he struggles with the lip reading. But now with Live Transcribe, he still uses lip reading, but when he's struggling, he can glance down at the transcript, and see what it was. It's changed his life because he works for Google, and he used to have to book a human interpreter, what they call a CART. That's something, something Real-Time Transcription or whatever. He'd have to book ahead of time for a meeting to have an interpreter online. But now he has an interpreter in his pocket everywhere he goes, and he says it's totally changed his life. We basically did it for him and by him. But it's had that same effect on so many people now. It's really rewarding to see the impact and the kinds of feedback that we get on that.
ZIERLER: Dick, what are the most important professional societies for you, either by journals, by conferences that you go to? What's been key?
LYON: Interesting question. I'm a Fellow of the IEEE, and a Fellow of the ACM. But that doesn't mean those organizations are particularly important to me. In the older days, the IEEE was super important because that's where all the conferences were that I went to, to learn the field, like the International Conference on Acoustics, Speech, and Signal Processing that I used to go to every year from 1978. But I don't go to their conferences much anymore. They've gotten to be so big that I have a hard time getting anything out of them. In the hearing field, what's been really super is an organization called the Association for Research in Otolaryngology, the ARO. They have a thing they call the Mid-Winter Meeting. They have two meetings a year, and one of them is kind of like ear, nose, and throat surgeons, and the other one is more about hearing. That's been the place where I've met and gotten to know most of the people in the hearing research field. Another one, the Acoustical Society of America has also been useful in that way. But, again, acoustics is such a huge field that it's a dilute conference. The hearing psychophysics and physiology people there is one small piece of it. It's not as concentrated as the ARO in terms of the interactions you get by going to a meeting, but it is a good place to publish. I've got a couple papers now in the Journal of the Acoustical Society of America, which is a prestigious place for speech and hearing papers.
ZIERLER: Dick, not finishing with the PhD, not becoming a professor, if that's something that you were considering, what mentorship opportunities have you had, both in academia and in industry?
LYON: Oh yeah, I've had plenty. I've served on PhD committees, even though I don't have a PhD myself. I was on the committee with a number of Carver's students, and I think maybe also some not-Carver students at Caltech. I've been on committees at Berkeley, Stanford, Western Sydney University, Imperial College London. I feel like I've gotten some of the perks and influence channels of being a faculty PhD without actually going that route, as a Caltech faculty non-PhD. It's a little unusual for research labs to hire non-PhDs. In Google, there aren't that many of us in research that don't have PhDs. But my boss for many years, Jay Yagnik, also doesn't have a PhD, and so that's cool.
ZIERLER: Dick, in all the ways that you've been recognized and honored for your work, are there any awards that are most personally meaningful or special to you?
LYON: Yeah, I think so. The Royal Photographic Society gave me and Carver and Dick Merrill the Progress Medal for our work on developing the Foveon X3 technology. I was the one who got to go to London, and go to the ceremony, and receive the medals for us. They gave us these three medals in boxes, big chunks. I didn't realize that around the edge of the metal, they had engraved our names, and I handed them out to Dick Merrill and Carver Mead, and we each got somebody else's medal, so that's kind of cool. [laugh]
ZIERLER: Dick, of all the work that you've been involved in, what has been most fun? A particular project, a particular collaboration, just simply put, what has been the most fun for you?
LYON: I'd say definitely the early years of Foveon, developing cameras, because of all the new stuff I got to learn, and the intensity of the environment, knowing that we were on the one hand creating stuff very new, and on the other hand we had to make and sell product, and make customers happy. When we did that, we got stuff out there that made customers very happy. That was so rewarding, and the process of getting to it was so much fun.
ZIERLER: Finally, Dick, last question, looking to the future. You were at 50%. Now you're back up to 70%. What's a timeline for you, do you have one, and what do you want to accomplish for whatever that timeline is?
LYON: The timeline's not clear, but I want to get the Australian collaboration projects running better. We're still bringing in postdocs to do the real work there. I'd like to see them deliver some results. I don't necessarily have to stick around till it's finished and actually influencing hearing aids on the market, but I'd like to see that the results are going to work and it's headed that way before I retire. I've got some things left on the table at Google that I want to try to influence toward a finish as well.
ZIERLER: Dick, this has been a wonderful conversation. I'm so glad we were able to do this. I'd like to thank you so much.
LYON: It's been a pleasure, David. Thank you.
[END]
Richard F. Lyon (BS '74), Principal Research Scientist, Google