Victor Wei Du, born guarding the capital

Early Life

My name is Victor Zue, and I am the youngest of four children of my parents. The oldest was named Helen, the second was named Deanna, and the third was named Juliet. In Chinese culture, it is the male children that carry the family surname. In my family the burden was on my shoulders.

Below is a photo of my family with my grandmother on my mother’s side. Left-to-right, the standing people are my oldest sister Helen, my father, and my mother. The front row are my third sister Juliet, me, my grandmother, and my second sister Deanna.

I was born on February 19, 1946, in Chengdu, the capital of Sichuan Province. To understand the story of me and my family, one needs to understand the history at that point. The Sino-Japanese War was unfolding from 1937 to 1945 resulting in the death of over 10 million Chinese civilians. On top of this, the forces of the Communist Party were rapidly mobilizing in preparation for overthrowing the ruling government.

My parents lived in Shanghai at the time. On October 10th, my father (a high school teacher) gathered a small number of his students, and went to the tallest building in Shanghai and put the Chinese flag on the top of that building. As a consequence, there was a bounty on his head and he had to leave Shanghai. They only had one daughter, Helen, at the time. What they did was they got to Burma. At that time, the United States had not declared war against Japan. But they had built a place where they could construct airplanes in Burma. My father (a very good manager) went there to start building warplanes for China to fight the Japanese.

Originally, it was a regional war between China and Japan, and until Pearl Harbor, it wasn’t considered the Second World War. And my father was a leader of the underground work against the Japanese.

It turns out that I have three sisters, and our names all have one Chinese character indicating where we were born. Helen (Hai Lun) — Hai means Shanghai. My second sister’s name is Deanna. Deanna is Burma. The third sister and I were both born in Sichuan. So Juliet’s name is related to Chengdu. And my name is Wei Du; the Du is for where we lived in Chengdu.

One story I have heard my mother tell me involves their travels during the war. My mother had taken Helen, the only daughter, trying to get to Chengdu. My father was coming from the Southern part. There was only one road going all the way to Chengdu. And she stayed at a farm along that road. She hung out with Helen at the roadside for days. Whenever a car passed, she’d call my father’s name. It went on for days. It’s a miracle they met up.

There were some other, nearly unbelievable, harrowing stories. For example, according to my memory from my mom’s recounting, during their travels, they were stopping in a village to grab a bite to eat, and all of a sudden, Japanese bombs were falling on the village. My mother claims that one bomb landed on a family at a nearby table, and they were all severely injured or killed. Remarkably, my parents and Helen were unscathed.

My father went to Burma where Deanna, his second child, was born, and then he went to Sichuan, because he was trying to get to the wartime capital, Chongqing. Instead, we were in a very nearby city called Chengdu. And that’s where my name comes in, Wei Du; the Du is for the town, and Wei Du means guarding the capital. They lived through the remainder of the war in Chengdu. Eventually, the war was over. During the war, the mainland Chinese government and the communist government decided to shake hands and fight the common enemy, the Japanese. The civil war between the communists and the nationalists was on hold.

But the minute the Japanese war was over, the communists and the nationalists restarted the civil war. The communists were able to make great advances. Our family went back to Shanghai. And my mother’s mother, my grandmother, was there. We picked up my grandmother and then we took a boat and moved to Taiwan. It was a sad story because, in Chinese custom, a mother is supposed to stay with the son. My mother has a younger brother who lives in Shanghai. We convinced Grandmother to come with the understanding that, once the internal war was over, she would return to live with her son, i.e., my uncle. But it never happened. In fact, she never saw him again before she passed away.

Teenage Life

My father, Charles, was a Chinese nationalist. That means he supported the Chinese government that existed before the Communist Party came into power. He decided to move to Taiwan when I was an infant, because the Communist Party would overthrow the Nationalist government in China, and the leaders of the Nationalist government, the Kuomintang, would relocate to Taiwan. At that time, the Nationalists and the Communists were still at war. As a result, the Nationalists had mandatory military service for every male. So, when I was in eighth grade, my father decided to leave Taiwan and move the family to Hong Kong.

At that time, this guy Run-Run Shaw (who owns a movie production studio in the Kowloon Clear Water Bay area) heard that my father was a good manager. So he said “Hey, Charlie. I’m setting up a movie production. Why don’t you come work for me?” In fact, Deanna was given a job there as well. Every day Juliet and I had to commute to school by bus, which could sometimes take over an hour, maybe two hours, each way, depending on traffic.

Going to America

All I wanted to do at the end of that high school year was to return as a foreign student to go to a university in Taiwan. My father, on the other hand, said, “You’re not going back to Taiwan. You are going to go to America.”

Florida Years

In Chinese history, the father speaks and everybody follows. He said, “You are waiting for admission to the University of Florida.” Florida, because my older sister, Helen, went to school there. She studied horticulture, in particular citrus production. So there were two choices for her: California or Florida. She picked Florida. My second sister, Deanna, always stayed with our parents and never attended college. The third one (Juliet) went to the University of Florida in Gainesville because Helen was there. Likewise, I went to Florida. This is the immigrant mentality, right? You go to a known place and try to go where you have relatives.

I arrived in Florida on Labor Day 1965. I attended the University of Florida. I remember moving into a dorm. It was three people per room. The first person I met was Rob Moore, who turned out to be my best friend throughout my life.

One thing about Rob is that he took me under his wing. Every time I said something wrong, he would correct me. It was, at first, embarrassing, but then I realized he was doing this as a favor to me. He wanted me to be assimilated into the culture.

After I finished my undergraduate degree, I continued with my Master’s degree under the advice of Prof. Arnold Paige, who convinced me to go to MIT and work with Prof. Ken Stevens, who had been his advisor. I applied for MIT admission but was not accepted. I was asked by the graduate admissions officer if I would like to be put on the waiting list, and I said “yes.” That was the first example of many happenings in my life where the second time is often a charm.

At the University of Florida, I was dating one of the very few female Chinese undergraduates from Taiwan. Her name was Su Lin. It never occurred to me that I would date an AMERICAN.

To give my sister Helen credit, she was trying her very best to talk me out of my first marriage. But I did not listen. When I decided to go North to Boston, it was not appropriate for an unmarried couple to travel many miles together, so we decided to get married. The marriage didn’t last very long…

The difficulty of going to MIT was just unbelievable from my past experience. I studied very hard, and one of the first courses I took was called digital signal processing. We were in a classroom, maybe 120, 130 students. There was only one woman in the class, and she was pregnant! She was a special student because she was working at Lincoln Labs in Lexington. A benefit Lincoln Labs offered was that you could actually take a course on campus if you wanted to, with full compensation if you got a B or better.

And the amazing thing was that the professor, Alan Oppenheim, was a very Socratic person who would like to ask questions to break up the lecture. Periodically, whenever Professor Oppenheim would ask a question, the students would be petrified. They would all start looking at their notes, finding something very interesting to look at. Then he would say, “Stephanie,” and she would answer. It happened repeatedly. First of all, she was the only woman. Secondly, she was pregnant with Mike!

Of course, that was Stephanie, who I fell in love with a few years later.

I did well enough in the class that Professor Oppenheim said, “I’ll take you on as a student.” On the MIT campus, the computing capability wasn’t nearly as good as at the MIT Lincoln Lab, which is out in Lexington. Professor Oppenheim said “Why don’t I try to see if I can get you a desk to be a student using their computers to do your studies?” He was able to convince the group leader, Ben Gold, to let me study out there. So I had an office, and my office mate was Stephanie. That was the first time we actually got to know each other.

Serendipity.

That was a rocky relationship because she was kind, and she loved drinking coffee. But the minute she got pregnant, she couldn’t stand even the smell of coffee. And me, I had a percolator. So all day, I had the percolator going, drinking coffee. I noticed that every time I started making coffee, she would make a beeline to the bathroom. Later I realized she was rushing there to throw up. She never told me, never said, “This bothers me. Would you mind not?” She just suffered through it. Isn’t this amazing? She’s just that kind of person.

It’s interesting because there’s a very famous researcher, his name was Bishnu Atal, who pioneered a particular way of doing signal processing. It’s called linear prediction. Without going into the detail, anyway, Bishnu came to Lincoln Lab to give a talk about this new way of doing signal processing, which could have implications for coding, reducing the bit rate for transmission for phones, and also for speech recognition, because you could get a small number of parameters to represent the speech. He gave a talk to about two or three hundred people, and at the end, unbeknownst to each other, both of us were smitten by the work. After the talk, before we were even office mates, we checked out the only two books on this theory. Two books, and then we both studied and tried to do the work. Her boss was Ben Gold and my professor was Al Oppenheim. We were starting to work on the same thing independently. So they called for a meeting of the four of us, and they had to decide who would do this and who would do the old thing. Stephanie had been working on this method—again, the name may not mean anything—homomorphic analysis. So the four of us had a meeting with Ben Gold, the group leader, and they said, “We can’t have both of you working on this.” And they both agreed that Victor should do it because Stephanie already had another project, and also, I needed a thesis. So Ben Gold said, “I think Victor should do this.” I saw daggers coming out of her eyes, and she said, “I quit!”

It was pretty tough. We were not office mates yet. I was in a bullpen of graduate students. I still remember she and Bill McCandless would, on Thanksgiving, invite all the graduate students to their home for dinner, except me.

Eventually, I think she thought I did a good job pursuing that work, and she forgave me.

We were married but not to each other. In my case, the writing was on the wall. It was clear it didn’t work out. In her case it was not. She subsequently had a second child, Gregory.

We lived together before we married and made our home in Concord, Massachusetts in a farmhouse shared with another unmarried couple, Phil Henshaw and Leonie. He was going through school at MIT, and she was working at Lincoln Labs. It’s almost like the same thing as we were going through. Leonie had two sons from a previous marriage. Shortly after we got married, we decided to have some kids of our own.

To show how special Stephanie was and is, in Chinese tradition, the parents stay with the son’s family. And there was only one son and three daughters in my parents’ case. When we finally bought the house on Glengarry Street in Winchester, Stephanie said they should come live with us.

So she invited them to live with us. At the time, it was two young kids not related to me, and my parents, all living together with the two of us.

My English was good enough that she did not need to learn Chinese. My father had worked in the motion picture business and his English was good. But my mother did not speak much English. So Stephanie decided to learn Chinese, which she speaks to this day, because my mother was living with us, and she wanted to be able to communicate with my mom.

This is the house that Stephanie and I bought together before we were married on Glengarry Street in Winchester, Massachusetts. It was here that we decided to get married and embrace a life together with her two kids, Mike and Greg, and my two parents.

In that same house, Stephanie gave birth to two more children, Tim (1977) and Cory (1982), both by natural childbirth with no drugs and assisted by Mara White and two other midwives.

Professional Journey at MIT

I completed my doctorate in 1976, but even before I did, a number of universities were trying to invite me to join their faculty. I had not taken any of those offers seriously, because Stephanie had begun to get her Doctor of Science degree at MIT. So I got my degree at about the same time that she started pursuing her Ph.D.

Back in the old days, when people thought about speech recognition, they thought of it as dictation. You speak, and words come out on the screen. The thing that Stephanie and I pioneered is the idea of speech understanding. The idea is turning speech into words, and words into meaning. Once you have that, you can have a dialogue with the machine, which we now take for granted.

I was more into the signaling part. She was more into the language part. And both of us were working on the dialogue part. Because it’s one thing to say, you take the signal, turn it into words, and turn it into meaning. But then you still need to go one exchange at a time. Ultimately, you want to have a conversation with the computer. So that was the thing people saw that set our work apart from that of others in our field. I stayed at MIT as a research scientist, the lowest rank of researcher. I was not offered a faculty position. So there’s faculty, and then there are researchers. And the faculty are in charge.

The ranks for professors are, from highest to lowest, Professor, Associate Professor, Assistant Professor. Then the research ranks: Senior Research Scientist, Principal Research Scientist, and Research Scientist. So I took this position here. I was never offered a faculty position at MIT initially.

I continued to work at MIT while Stephanie pursued her Ph.D.

Leadership and Challenges at MIT

The name of the person that influenced our life the most was Michael Dertouzos, who was the lab director of LCS, the Laboratory for Computer Science. He, for some reason, saw something in me. He was a flamboyant Greek person, about 6’10” or something like that. One day he just called me up and said, “Hey. You don’t know me, but I’d like to have lunch with you.” So he came over in his BMW and we zoomed all the way down to the Hyatt Regency in Cambridge. While he was driving, he said, “I have no hidden agenda. I want you to come to LCS.”

I was smitten. I came back and told Stephanie, “Hey, we have a chance to move to another lab where they’re actually doing computer science rather than signal processing.” So, the next thing you know, we moved across the street to LCS. Over time, Michael saw enough potential in me that he decided to make me the deputy director.

For one thing, it’s always good to have a boss who believes in you. Therefore, you have the confidence that, if you continue to do the work, then your career is going to flourish. And so, after we were there for a little while, he decided—he tried to put me on the faculty. He tried. The first time, it wasn’t successful. It was quite difficult. But then he didn’t give up. He tried again. The second time, it happened.

He knew I was a good scientist and engineer, and he took me under his wing in different ways. For example, in management. He was the laboratory director of LCS. Shortly after we moved, he assigned me as the assistant director of LCS. He thought, within LCS, there are maybe 10 or 15 different groups. The assistant director and he were basically managing these groups.

The most important thing I learned from him is love. You have to make the members of the lab feel involved and committed. Therefore, if they feel loved, then they will excel beyond the ability they think they have.

Back then, it was the common belief that it was impossible for a machine to capture a voice and actually understand what was being said. It couldn’t translate these sounds into words. The signal was specifically coded for the human ear, and that code would be impossible to crack. You think about all the systems today that are voice activated, that rely on machines understanding people. And it’s just wild that people didn’t think it was possible. Back then, the way to build a machine that could accomplish some particular task was to understand how a human does it, and then to map the mechanics of that processing from the human to the machine.

You start by capturing the raw auditory signals, and those are captured and displayed in a format called a spectrogram. A spectrogram is basically a visual representation of an auditory sound, like speech. But this was where people got stuck. A human, they thought, could not look at a spectrogram and interpret it to be real human speech. So there’s no way a computer could figure it out. Through endless hours of studying these waveforms and banging my head against the wall, through years of research, I was the first person to demonstrate that a human could look at a spectrogram and interpret it. Not only was I able to develop this skill as an individual, I was able to codify it, mechanize it, and then build data sets and systems that would allow machines to do the same thing. In this way, I (with help from Stephanie) laid the foundation for modern spoken language systems. In part through my scientific breakthroughs, we eventually had tools and systems like speech transcription, interactive voice response systems, Siri, Alexa, Google Assistant, you name it.

One thing about me I’m rather proud of is the fact that I don’t like to take things as they are. If spectrograms cannot be read, then maybe it was the fact that those people who couldn’t do it didn’t know what to do. I have a little story to tell. It was much later, maybe within the next decade. At that time, one thing the world knew about spectrogram reading, or voiceprint, was that I was a pioneer for doing something like that.

There were some people at the time who said publicly that it could not be done. One of the keynote speakers at a conference in Australia (his name is not important) built his talk around the idea that humans could not read spectrograms. He was of the opinion that this couldn’t be done because only very special decoders in your ears connected to your brain could interpret the signal. The idea that sound coming from your mouth could be understood by using your eyes to look at this display seemed impossible to him. At the beginning of his talk, he put up a spectrogram and said, “If Victor Zue were here, he would tell us what this says.” I was there, and I could not resist shouting out the sentence from the audience. His talk got off to a bad start!

Once we were settled in LCS, I started traveling back and forth to Taiwan and other parts of Asia (Japan, Korea, Singapore), to build research collaborations. Then in 2001, a lot of things happened during a particular trip I took to Taiwan. I told Michael that we might want to recruit some of the companies we were visiting to be sponsors of our laboratory. Immediately Michael said, “Yes. Let’s start something called Project Orient.” We started by doing some work with NTT and other companies in Japan. Next, I started to work on Taiwan, making a number of trips together with Michael, which started the flow of funds to LCS. Then I was asked by him to be responsible for that part of the Asian Project Orient.

Michael was quiet about the fact that he was suffering from a strange condition, and he would disappear for weeks at a time. He didn’t tell anybody that he had this incurable disease called polychondritis. I was on one of those trips, and I had just checked into my hotel in Hong Kong when the phone rang, and the person from MIT said, “You’d better come back home because Michael has just died.” It was a big shock.

Without even unpacking my suitcase, I turned around, got to the airport, and flew back. At the time, the MIT president was Chuck Vest. When I returned, I was having lunch at the faculty club, and Chuck just came over to the table and put his hand on my shoulder and said, “Hey. You’re going to be the next director of LCS.” They decided to bypass the search process. And I was just petrified.

This was another key moment in my life. One, you’re losing a huge friend, colleague, supporter, and mentor. And two, you’re stepping into an immense job. I had seen Michael do the work before, but it wasn’t like there were any handover notes.

My job since then changed dramatically. I used to enjoy what I was doing. I didn’t overreach. But in this case, all of a sudden there was nobody I could ask for help. I had questions and I had to make decisions based on not being able to sleep at night and trying to figure out the next day’s schedule. That’s when I started the process with a little pad. I put it in my pocket, and every day I wrote down ten things to try to get done that day. And this practice has gone on since then. Because you have so many different asks and needs and emails and calls and meetings and everything else. You need to know what are the top 10 most important things that you’re going to try to make progress on.

At the end of the day, I would find all the things I had done and check them off. The things I didn’t finish were written on the next day’s list, and so forth.

I would say that probably the stress of that job contributed most to my deteriorating health. I started having high blood pressure. I started having all kinds of problems with my heart. In fact, I was a mess. I had problems with diabetes. They all happened in rapid succession. If I had continued as director of the Lab for Computer Science (LCS), it might have been different. But it turns out that the two labs – Laboratory for Computer Science (LCS) and the Artificial Intelligence Laboratory (AI Lab) were on a path towards a merger. At the time, MIT central administration and also the lab director, Rodney Brooks of the AI Lab, had a conversation with me. Rodney said, “Come to my house for dinner.” We agreed that he would be the founding director of the new joined laboratory, CSAIL (Computer Science and Artificial Intelligence Laboratory), and I would be the co-director of CSAIL. That meant I didn’t necessarily report to him, but maybe when we were together, one (him) was a little higher rank than the other. He was also very good at giving talks to the outside world, which was not my cup of tea. I was always the person who liked to get practical things done, but not pontificate in terms of giving talks that somehow try to get people excited.

Within one or two years, Rodney began to have really serious interests in companies outside of MIT. He started a company called iRobot, iRobot was the creator of the Roomba. It made quite a splash at the time because it was arguably the first robot that you could buy for home use. The world’s first commercial autonomous vacuum cleaner. When we were together as director and co-director, Rod made sure the world knew about CSAIL. And I made sure that internally the train runs on time. Some of its members were Turing Award winners, including Tim Berners Lee, the inventor of the World Wide Web, and Ron Rivest, the R in the RSA cryptosystem. The Turing Award is the highest honor possible in Computer Science, because it is not one of the fields that host a Nobel Prize.

Eventually, mainly for health reasons, I decided that I didn’t want to be director anymore. So I announced that in six months’ time, I was going to step down.

Here is one photo of the party the lab organized to honor me when I stepped down. There is an outdoor amphitheater called the Michael Dertouzos Amphitheater, named in his honor. The funny thing was that I was known among the professors for wearing red socks. And then everybody that day wore red socks. All the professors and researchers were sitting in the amphitheater and they took a picture with me and Stephanie sitting in the middle.

Barbara Liskov is the woman in the front on the far left who didn’t seem to take to the red socks idea!

A Life of Contentment and Family

I now live with Stephanie, my wife for fifty years, on one of the Hawaiian islands named Kauai, a place we fell in love with many years ago. Our four children have now given us eleven wonderful grandchildren. Looking back on it all, you can see this man who has shaken hands with many of the leaders and luminaries of our generation. Yet when you look at some of the key steps in my life, it’s been a story of second chances. It took me two chances to get into MIT, two chances to become a Professor, two chances to get tenure, and two chances to become a lab director. And, of course, two chances to find the love of my life, Stephanie Seneff.

Do I have any words to share for those of you who are still on your first chance, still trying to find your way and your footing in life, particularly when things fall apart, when you don’t get the thing that you’ve been fighting and struggling and dreaming about? I see it in two ways. One, sometimes it’s just not the right time. Maybe you’re destined to take a certain path, to go down a certain route, to achieve the thing that you want, but it’s going to take another year or another 10 years, but you’ll get there. But then the second way of looking at it is sometimes you do make a mistake, but life is long and it’s full of second chances to get things right.

Now I have the idea of being near a peaceful lake which is quiet and without any ripples.

And, if that were the way I look at my life, then I would say, imagine such a tranquil place where you drop a drop of water, in there and, metaphorically, when you drop water into a quiet and peaceful lake, it makes a bunch of concentric circles. And in the middle of these concentric circles, there’s water.

In my life, in the middle it is, undoubtedly, Stephanie. And the next circle, will be my family. And most of the things that I think about have to do with people, and especially it has to do with people that are close to me.

I think some people may look at my life and say, “Boy, you have accomplished quite a bit for a Chinese person born in Sichuan and eventually got to some places where most people would say it’s pretty impressive.”

I’m content, and when you have a good partner, your life just becomes fabulous.