Stephen Jordan
Stephen Jordan
Principal Research Manager, Microsoft and
Adjunct Associate Professor, University of Maryland
January 14, 2022
DAVID ZIERLER: This is David Zierler, Director of the Caltech Heritage Project. It is Friday, January 14th, 2022. I am so happy to be here with Dr. Stephen Jordan. Stephen, thank you for joining me today.
STEPHEN JORDAN: Thanks for inviting me.
ZIERLER: Stephen, to start, would you tell me, please, your current title and institutional affiliation?
JORDAN: I now work at Microsoft and my title is Principal Research Manager. I also have an adjunct affiliation at the University of Maryland.
ZIERLER: What's the affiliation there? What do you do at UMD?
JORDAN: My affiliation is with the University of Maryland Institute for Advanced Computer Studies. I have one PhD student who's finishing up that I advise remotely. I picked up an adjunct faculty affiliation while I was an employee of the National Institute of Standards and Technology in Gaithersburg, Maryland. Prior to when I joined NIST in 2011, they had already founded the Joint Quantum Institute or JQI, run between the University of Maryland and NIST, which was focused on the physics side of quantum information. Shortly after I arrived at NIST, they founded a second institute on the same model, which I helped to name. We call it QuICS, the Joint Center for Quantum Information and Computer Science, which, as you may guess, is more focused on the CS side. Then when I came to Microsoft, I relinquished my formal affiliations with NIST and QuICS but retained my adjunct affiliation. This allows me to supervise PhD students in physics and computer science at the University of Maryland. When I came here, I asked my students what their preference was, and they all gave the same answer, which was that they would prefer to continue along the same path that they were already on. So, I've been supervising them remotely since 2018, and they've been graduating one-by-one, and I now have one left.
ZIERLER: Stephen, beyond just having an enjoyable time staying connected to academia, in what ways is the appointment at Maryland useful for your research?
JORDAN: I would say it keeps me a little bit more broadly focused than I might otherwise become. Since joining Microsoft, I've followed a path that I did not necessarily anticipate, and essentially became enticed by a much more applied kind of work than I had done in the past, which includes a lot of work on classical algorithms for solving industrial optimization problems. But I always have at least two meetings a week with my collaborators at University of Maryland that keep me from losing touch with the foundational research side of things.
ZIERLER: Stephen, is that to say that, ironically in some ways, the research in quantum computers might actually yield benefits for classical computers?
JORDAN: Yes. There are not so many changes in your life that you can pinpoint to a specific moment. But one of them for me was a specific day, probably around 2015, when I was working at my office on the University of Maryland campus. I was carrying out some computer simulations of adiabatic quantum algorithms. Adiabatic quantum algorithms were first proposed—well, there's a little ambiguity, as is often the case, about who first proposed things because there are few related ideas coming out at roughly the same time. But, anyway, one of the originators of this was Eddie Farhi, who was my PhD advisor at MIT. He proposed that adiabatic quantum processes could be used to solve optimization problems.
People got excited about this because, first of all, you could use a lot of nice techniques from theoretical physics to analyze these algorithms. But it was also a tantalizing subject because no one could really quite answer the questions that they really wanted to answer, and so it always seemed like the deeper understanding was right around the corner, which is very motivating. Also, a lot of people said that non-convex optimization problems are of great economic importance. This economic importance seemed sort of generally plausible to me, although I didn't have much direct insight into it at the time. Now, in my role at Microsoft, I get to see a lot more detail about applications. Anyway, a lot of people say that the possible advantage of adiabatic algorithms for solving optimization problems comes from quantum tunneling, which is different from, say, Shor's algorithm, where the advantage comes from interference phenomena. It's fairly clear that you cannot efficiently simulate the interference phenomena that underlie Shor's algorithm using classical computers. But it's less clear in the case of tunneling. Now, none of these are rigorously proven statements but this is my sense of things which I think a lot of people share. I and some collaborators were trying to prove some theorems about this. Basically, what I was trying to do is show that a certain class of these adiabatic algorithms could be efficiently simulated by classical computers, and therefore could not yield exponential speed ups analogous to Shor's algorithm. Now, the theorems didn't work out. We found a counterexample. But the counterexample was pretty contrived, so it still left open the possibility that these classical simulations worked well in practice most of the time. So, we decided to run some computer simulations to find out. We needed some optimization problems to run the simulations on, so I did an online search using the first keywords that popped into my mind about where to find such problems. I typed in, "MAX SAT benchmark," or something like this. The first thing that came up was a set of benchmark instances of MAX SAT problems that came from an annual contest to see who could build the most efficient solver for MAX SAT. So, I said, OK, well, that's as good of a place as any to get some test cases, and I downloaded these, and I simulated an adiabatic process to solve these problems. To my surprise, I happened to notice that our simulation of the adiabatic optimizer solved the MAX SAT problem faster than the winning entry from the most recent contest. I remember being pretty excited about this, and popping out of my office, and running down the hallway and telling people about it. At that point I thought we've stumbled onto something much more interesting than what we were actually looking for. So, we just took a left turn on our project, and started investigating these simulations of quantum algorithms as optimizers unto themselves, rather than as a mechanism for understanding adiabatic algorithms.
That's actually now what I focus a lot of my time on here at Microsoft. I came to Microsoft in 2018 in February, and I wasn't completely sure what direction I would take upon arriving here. But I think my casual assumption was that I would continue on a fairly academic-style, relatively pure quantum algorithms research path, similar to what I had been primarily focused on at NIST / University of Maryland. But, also around that time, the corporate vice-president in charge of the quantum effort at Microsoft at the time, Todd Holmdahl, decided that we should start to engage with real customers, and learn more about what are the problems with high business impact that are really important, and that also are compute bottlenecked, as opposed to ones where the limiting factor is data quality, or input-output bandwidth or latency, or user interface, or any of the many things for which quantum computing offers no possible advantage. We were looking for economically important problems where computation's really the limiting factor.
So, the business development director at the time, Julie Love, happened to walk past my office—maybe not so coincidentally. Maybe she knew that I was new and could be easily steered, having not yet been drawn into too many projects yet. She walked by and asked me if I wanted to join some of these meetings with senior level technical people from pharmaceuticals and power utilities and logistics companies and all these different places, and just talk to them, and learn about what their computational problems were. I thought that sounded very interesting, and so that's what I did. What we found was that the specific nature of the problems varied between industries but there were very consistent patterns that the most common problems were machine-learning problems and non-convex optimization problems. Every once in a while, you'd see something that would be a simulation problem or a Monte Carlo sampling problem. But the ones you would see over and over and over were machine learning / statistical inference and optimization. A lot of these companies were looking to do engagements, and so we thought, well, there are two kinds of engagements you can do. One is you can work with us on research projects about quantum computing. You can't currently—as of 2018 when we started these engagements or as of now in 2022—you can't currently use quantum computing to solve any actual industrial problems that you can't solve on your laptop. But we could use these things that we had discovered about simulating tunneling processes classically, and so we took to calling that stuff quantum-inspired optimization. That was the second kind of engagement, and that effort grew and grew.
Microsoft, unlike many other companies, treats the reporting structure separately from career advancement and thus gives people room for advancement without ever having to necessarily climb the org chart. They can climb the pay grades without climbing the chain of command. I am in what is considered a technical leadership position. I lead a team of applied researchers who are working basically full-time on quantum-inspired optimization, and applying it to improve efficiencies of large-scale processes, both internal to Microsoft and for various external customers.
ZIERLER: Stephen, I'm curious if we can compare the research that's being done at Microsoft looking back 50 or 60 years to Bell Labs, for example, where there was really a culture of fundamental science that didn't have anything to do or didn't need to have anything to do with the corporate bottom dollar. For you, is there a research culture at Microsoft that's promoting and supporting fundamental research without it necessarily needing to be articulated or connected to some business plan at Microsoft, or is really everything that's being done in the quantum world somehow related to where Microsoft might see a market or a business opportunity?
JORDAN: Well, in general, I would say that Microsoft Research definitely does support fundamental research. There are people there who write very foundational computer science papers in pretty much—well, not quite every area of computer science but a wide swath of it. So, if you compared it to a major and first-rate computer science department at a university like, say, UC Berkeley or someplace like that, and you looked at Microsoft Research, it would look very similar in terms of the list of topics, the type of people who work there, what their goals and motivations are, and so on. I would say at this point, regarding the quantum group specifically, we're deeply rooted in that world but we are actually pretty focused now on bridging the gap between fundamental research results and applications. We want to cross this barrier that's very hard to cross to get fundamental research results transitioned into real application. That's actually now the part that I'm focused on. People sometimes call it the Valley of Death. Fundamental research happens on one side, and incremental progress on existing technologies occurs on the other side. But to get something to jump from the fundamental research side to the applied technology side is really hard. I'm trying to learn as much as I can about how to do that right now.
ZIERLER: To the extent that we can think about applications of quantum computers, what aspect of the work at Microsoft is devoted internally, in other words, to making things at Microsoft, and what is devoted to Microsoft's customers or clients in terms of their needs?
JORDAN: I would say at present, the best idea out there for getting real value from quantum computers for problems that will affect human life, and generate economic growth, and cure disease, whatever it is, it's really computational chemistry. Microsoft is not directly in the business of quantum chemistry. We're not a pharmaceutical company. We're not a chemical manufacturer. We by ourselves our not going to develop new catalysts or new artificial flavors or whatever the case may be. Really, I would say, for the fully quantum technologies, the applications we're focused on are outward-facing; things where we would partner with companies that are in the business of doing things that are chemical or materials related, and have deep history and expertise in that. As far as the quantum inspired optimization, which ultimately are classical algorithms but have their roots in quantum research, that's focused both ways. We have large-scale industrial optimization-type problems internally, such as around how we operate our data centers, and we have already seen big wins from applying quantum-inspired optimization techniques to those problems.
ZIERLER: Stephen, without going, of course, into any sensitive details, one obvious concern with quantum computing is that it could be a big problem for internet security, for cryptography. So, in what ways is quantum computation a tool to protect the problem that it might create? In other words, how might quantum computers solve the problem of cryptography in a post-quantum world?
JORDAN: That's a good question. Quantum computers have an exponential scaling advantage for solving discrete logarithm and factoring problems, and so all of the public key cryptosystems currently in widespread use are broken by this. There are two kinds of cryptosystems: public key and private key, or, as they're also called, asymmetric and symmetric cryptography, respectively. Private key cryptography is not substantially weakened by quantum computing. Grover's algorithm can be applied, but if it ever becomes necessary, one could protect against this by using only slightly larger key sizes. The public key cryptosystems in widespread use are mainly RSA which gets broken by Shor's factoring algorithm and Diffie-Hellman which gets broken by Shor's discrete logarithm algorithm, which can be applied not only to integers but also to other groups such as arising from elliptic curves.
So, the question is what do you replace these public-key cryptosystems with? Occasionally, you may hear it argued that quantum technology provides not only the source of this problem but also the solution because there's something called quantum key distribution, or QKD, which uses quantum states of photons sent over optical fiber such that any eavesdropping can always be detected. But I don't particularly agree with this view because there are a lot of different things to use cryptography for. There's encrypting things that you store on a hard drive. There's authentication- proving you are who you say you are. There's transmitting messages that you want readable by a list of users, and not others. There's all sorts of different things you want to do with encryption. The ability to detect eavesdropping doesn't really give you all that. Furthermore, quantum key distribution requires special hardware, so it's not just a different firmware upgrade that you can put onto your router or something. You have to lay special fibers, usually, or at least they have to be dedicated, and it's point-to-point communication. It's also limited to a range of a few hundred kilometers, so it's not a very convenient technology to use.
Rather than QKD, in my opinion, one should almost always use post quantum cryptography. Post quantum cryptography consists of conventional software-based cryptosystems. They just base their security on other hard computational problems besides factoring and discrete logarithms. And nobody has found quantum algorithms to efficiently solve these hard problems.
People have actually known for decades of alternative cryptosystems besides the ones that use discrete logarithms and factoring as the foundation of their security. For example, lattice-based cryptosystems have been studied since the ‘90s. No one's found any serious security holes in them— with the exception of one of the signature schemes. So there are ways you can just do a software or firmware upgrade to protect your data against quantum attack, and you don't need any new hardware. You don't need to change your fundamental ways of engineering the big picture. It's just all the specific details inside your encryption module get written over with a different scheme. That's post-quantum cryptography, and I think that is realistically the solution that's going to be adopted in 99% of cases to secure your data and communications against quantum attack.
One key thing is you need the post quantum cryptosystems to be carefully scrutinized. The only way to know that a cryptosystem is secure is to have many experts pore over it and try to find flaws that leave it open to attack. Now we have to check for attacks both by classical computers and quantum computers. The US National Institute of Standards and Technology, where I used to work, is coordinating this process. Many organizations have submitted their proposals and helped to vet other proposals. This includes both academic institutions and corporations, including Microsoft.
The process works like this: initially there are many proposed cryptosystems. The flawed ones get eliminated. Then further scrutiny from the global cryptographic community gets concentrated on those that remain, and so on. Eventually, at the end of this process you have proposals that have withstood a very substantial amount of scrutiny by a very large community of experts. These get recommended as standards. That's playing out now, and that should shake out over the next few years. Then we will have an established standard for what people should use for post-quantum cryptography. I think that will likely be a very adequate solution.
ZIERLER: Stephen, a fun question more germane to your research, what is the Quantum Algorithm Zoo, and how did that get started?
JORDAN: Well, the Quantum Algorithm Zoo is a website that I have been maintaining since about 2008 or 2009, and it's supposed to be a comprehensive repository of quantum algorithms that offer speed-up over classical algorithms. The criterion is you have some notion of your problem size, for example, in factoring, you say, OK, well, we want to factor in N digit numbers, so the size is N, or if it's a chemistry problem, maybe it's a molecule with N electrons in it. So, that's what your notion of size is—whatever. Then, as N grows, how does the run time of your algorithm go? Does it go like N squared? Does it go like two to the N? The criterion for quantum speed-up that I tried to adhere to in the Quantum Algorithm Zoo is that the scaling of the proposed quantum algorithm should be asymptotically, in the limit of N very large, better than the scaling of the fastest known classical algorithm. Every example of that, ideally, I would have documented in the Quantum Algorithm Zoo.
Of course, I'm always somewhat behind on updating this, and I never keep up with everything. It's especially hard now as the number of people involved in the quantum information community has grown exponentially, and the number of papers per year published in quantum information has grown exponentially. But that's the idea of it. The origin of it was writing my PhD thesis at MIT. Basically, my PhD thesis had a sandwich structure. In the middle, I just had a series of journal papers that I had already written and published. Those formed the central chapters. Then sandwiching those were an introductory chapter and a final chapter that were new material. The final chapter was some summaries and philosophical musings and speculations and so on, and the introductory chapter was a literature review on quantum algorithms. I decided that I would attempt to make it complete; that I would list every quantum algorithm. Then I thought, well, this could be nice as a web page, and then it could also be continuously updated. So I took that chapter of my thesis, and ran it through this little script that was popular at the time called LaTeX2HTML, which automatically converted this document into a web page, and I hosted it somewhere. Then, ever since then, I've updated it. I had to move it a couple times so the URL has changed I think twice. But now it's at quantumalgorithmzoo.org, and hopefully that's its final URL, and people's bibliographies won't become broken links.
The name of the quantum algorithm zoo was stolen from Scott Aaronson. He made the Complexity Zoo first, which was a similarly comprehensive catalog of complexity classes. I asked him for permission to use his zoo terminology just to make sure he wouldn't be offended that I was stealing his turn of phrase, and he graciously granted me permission. So, it's been going ever since. Also, Victor Albert emailed me maybe a year or so ago, asking if he could borrow my turn of phrase because he wanted to make a quantum error correcting code zoo, which also is now a very nice web page that he maintains. So, one thing has led to another. Back in 2008, quantum information was relatively off the radar of most people. It was kind of an obscure field. Most universities weren't hiring anyone to do it. It was a little bit of an underdog in some ways, looked down upon for various reasons, and, of course, now all that's changed quite drastically. Anyway, I don't have a way of tracking the traffic on the quantum algorithm zoo web page but I think it has increased a lot since then.
ZIERLER: Well, Stephen, let's go back to graduate school to set the tone for your postdoc at Caltech. When you entered graduate school, this would've been what, 2003, 2004?
JORDAN: Yes, 2003, fall 2003 was when I started at MIT.
ZIERLER: Now, was quantum information, quantum science, was that already on your radar at that point?
JORDAN: Yes. I applied to graduate school, claiming that I would become a condensed matter theorist, and thinking in my mind that this was also indeed the most likely thing that I would do. But also in the back of my mind, I thought the second most likely outcome was that I would do quantum information. In my first year, I took Peter Shor's course in quantum algorithms and a few courses on condensed matter theory. And I liked the quantum algorithms side better, basically for two reasons. One is that it was easier for me to pick up. I was a little underprepared for some of the condensed matter theory courses, actually. Whereas quantum information was a little bit more accessible because it was using more discrete math, more linear algebra, and so on. It was accessible using more elementary mathematical language, but it was not any less intellectually stimulating because you still needed a lot of cleverness to think of new quantum algorithms, new quantum error correction schemes, and various other gimmicks.
But it was easier to get into it. It's kind of like a game of chess. Learning the rules of chess doesn't take very long. That's the case for quantum information to some degree compared to, say, high-energy physics or condensed matter physics where you have to build up deep layers of erudition to even reach the current interesting research questions. So, that was one thing I liked better. The other thing I liked was it seemed like it was a young field, and there were good prospects for discovering some of the really important foundational things that end up in the textbooks. Like, if you looked, you'd see that a lot of what Peter Shor was teaching in his class, was discovered five years prior by someone who you could go meet at a conference and talk to them. Whereas in the condensed matter theory course, you would be taught about something that Landau figured out in 1962 or something. So, it seemed to me that, statistically, the prospects were better in quantum information for making some discovery that ends up being a basic thing that goes in the textbooks and that everyone uses, and that was my big ambition at the time.
ZIERLER: Not that you would've known at the time, but looking back, was there an institutional organization at MIT devoted to quantum information like the IQI?
JORDAN: The quantum information efforts at MIT were not centralized in any such institution. You had Eddie Farhi who was in the Center for Theoretical Physics, which really was a center for theoretical high-energy physics. But he just decided personally, as an individual, to change subjects. You had Peter Shor in the applied mathematics department. You had Seth Lloyd in the mechanical engineering department, oddly enough. You also had Isaac Chuang and Jeff Shapiro who ran the Research Laboratory of Electronics, which was affiliated with electrical engineering. RLE was somewhat organized and systematic in their approach to quantum information. But, for the most part, quantum information at MIT was kind of scattered all about. In fact, another interesting thing is, in those times, if you looked at who were the faculty members at US universities who were doing quantum information, almost always they had been hired to do something else. Then after they got there and usually after they got tenure, they switched into quantum information.
ZIERLER: [laugh]
JORDAN: I think if you tried to apply to most universities at the time, saying, "Yes, I'm going to do quantum information research," it was a very hard road because physics departments would say, "Oh, well, it's not really physics. It's more like computer science. You don't really fit here." Computer science would say, "Oh, well, this is pretty remote from our concerns. Maybe you should try the physics department." A lot of people also thought quantum computers are kind of like science fiction. They thought of it the way you might think about space elevators or something.
ZIERLER: [laugh] Was Farhi your advisor?
JORDAN: Yes, yes. He became my PhD thesis advisor. After I took Peter Shor's course, I wrote a research paper. It wasn't my first research paper. Back when I was an undergraduate at Penn State, I did some undergraduate research, and I was a coauthor on two papers, one which was experimental physics studying superfluid helium, and one which was computational physics studying graphene. But after arriving at MIT, I became very inspired from taking Peter Shor's course, and I was thinking about quantum algorithms all the time, and I had an idea which I wrote up into a short paper, which is about estimating gradients on a quantum computer. I did not have an advisor yet. It was just something I did individually. Then I showed it to Peter, and he encouraged me to submit it to Physical Review Letters, which I did. I think initiating and carrying out a research project independently as a first-year student was a bit unusual, and I think that helped me land Farhi as an advisor. I was admitted into condensed matter theory, and he was in high-energy physics, so I was going across boundaries.
ZIERLER: What was Farhi working on at that point?
JORDAN: He was mostly working on adiabatic quantum computing, and secondarily on quantum walks, both of which are quite active areas of research still this many years later.
ZIERLER: Up the road, were you following Misha Lukin, what he was doing?
JORDAN: I was aware of his group but I never visited. I did visit Alán Aspuru-Guzik's group over at Harvard and collaborated with them on one journal paper.
ZIERLER: What was the process for you developing what would become your thesis research?
JORDAN: Well, I think I was influenced somewhat by Eddie Farhi's history. He was always thinking of alternative models of quantum computation. So, rather than just sticking with the standard way of thinking about things in terms of quantum circuits, he would use different physical inspirations for different frameworks for quantum computing. He would pursue these either as mechanisms of physical implementation or, more frequently in his case, as frameworks for coming up with new algorithms. So, that was definitely a theme in my research. But, mostly, I just kind of followed things where they led, and read journal papers. I read a lot of papers by Dorit Aharonov, which were also a heavy influence on me. Whatever ideas were spurred, I just tried to pursue them as I found them. One of my theories was that, when you first learn something, you think of all sorts of questions and tangents and things. I didn't want to waste that opportunity. I wanted to leave no stone unturned and look into all these little side paths and questions as I noticed them when I was initially learning new subjects, rather than trying to learn the subject first, and then go back and come up with research questions once I was up to speed. I don't know if this was a good system or not, but that was one of my principles that I used.
ZIERLER: Now, were you aware of IQI, what was happening there while you were a graduate student?
JORDAN: Yes. I didn't maybe know about IQI exactly, but I did know about John Preskill. I was certainly aware that he and his group were a major center, and I was aware, of course, of Alexei Kitaev. Eventually, I made a visit out to Caltech. At some point, I wrote a paper about error suppression methods for adiabatic quantum computers. Then I saw John Preskill at a conference and chatted with him at lunch, and told him about the main idea. He found it interesting. So, then, he invited me to come give a seminar at Caltech. That was probably about 2006 or 2007. Then later when I was graduating, I included Caltech in the list of places that I applied to for postdocs, and indeed went there.
ZIERLER: What were some of the conclusions of your thesis?
JORDAN: It was a bit of a hodgepodge, the title of which was Quantum Computation Beyond the Circuit Model. The unifying theme was exploration of different models of quantum computation, which was inspired by Eddie Farhi's work. But, as to the specifics, it was a fairly disjointed set of results. One was a mathematical result where I could show that there was this weird connection between a physical model of quantum computation called the one clean qubit model where you use very high-entropy, noisy states, and a mathematical issue which was estimating the Jones polynomials for certain classes of knots. There was just this weird connection between things that seem initially unrelated. You could make a mathematical precise equivalence between them. So, that was one thing in there. Then the rest of it was mostly about adiabatic stuff. On the physical side, I studied how to deal with errors for adiabatic computers. On the algorithm side, I studied some things about how to use certain more unlimited adiabatic processes to simulate more complicated ones efficiently using what are called perturbative gadgets. My thesis was mostly a compendium of the nuggets that I had discovered during my time in graduate school.
ZIERLER: Now, before you got to Caltech, you had a brief stint in Japan. Tell me about that.
JORDAN: Yes. Well, again, it arose from meeting someone at a conference. In this case, it was Sahel Ashhab. We got to talking about our research, and we had shared interests having to do with adiabatic quantum computing. Sahel worked as part of Franco Nori's group at Riken, which is a research institute in the outskirts of Tokyo, and so they invited me over there to do a stint over the summer of 2008. I thought it sounded like fun, so I agreed to do it. I went there under the premise that I was going to study things about error resilience of adiabatic quantum computers. But after I was there a few weeks, that project seemed to be stuck and going nowhere, so I just dropped it and worked on something completely different. Specifically, I worked on the continuation of my studies into link invariants, like Jones polynomials and so on, and their connection to quantum computers. But they seemed not to object to this really. It was quite a fun experience to live in Japan for two months, although I did not pick up any meaningful amount of Japanese—
ZIERLER: [laugh]
JORDAN: —so that was a limitation.
ZIERLER: Did you consider anywhere else besides Caltech for your postdoc?
JORDAN: Yes, I applied to many places, and had some options to pick from. I considered places both foreign and domestic. But Caltech ended up being my top choice.
ZIERLER: What were your initial impressions when you arrived in Pasadena?
JORDAN: Well, the group was very vibrant. It's unusual because it's sort of top-heavy. There were more postdocs than graduate students, at least at the time, which I think is quite nonstandard. So, that was a noteworthy thing. The people there, I really liked a lot. It was really a wonderful group of people. Also, the physical setting was extremely pleasant at Caltech, as I'm sure you know. I was unfamiliar with Southern California. The first time I had ever been there was that time I made a visit to Caltech to give a seminar, which was maybe about one year prior to moving there. It really seemed very foreign and exotic to me at the time. The vegetation and climate were very different, and even things like the way they pave the streets; it's with a different material than what they use on the East Coast. There were many such things that surprised me. The distribution of restaurant chains is completely different than on the East Coast, and I hadn't realized how foreign it would be in these respects. As to how foreign it is culturally, a lot of people say the culture is different in the northeast versus LA. I have no idea about that because, the fact is, I mostly just interacted with my little group of peers at IQIM. [laugh]
ZIERLER: [laugh] What was the game plan for you? In what ways did you want to expand on your thesis research, and in what ways were you open-ended in looking for new projects?
JORDAN: There was some of each. While I was there, I did continue some work following this thread having to do with topological invariants. I was intrigued by modern mathematics, and I had not been trained in it, and I kind of glamorized it. I wanted to be able to understand these exotic mathematical things that I'd heard some other people talking about which they understood and I didn't. I kind of envied their understanding of those things. So, that was probably the real motivation behind pursuing topological invariants and their connections to quantum computing. Then I also started a new line of research that I hadn't been working on at MIT, which was about simulating quantum field theories on quantum computers, which I think had more sound motivations. There were good reasons to study that question rather than just as an excuse to learn about some areas of mathematics. The quantum field theory project addressed a well-motivated scientific question—really two questions. If you go back to the origin of quantum computing, it started with the observation that there are certain processes in quantum mechanics that seem to require exponential resources on classical computers to simulate. So that was a hint that there might be more computational power there in quantum processes that could be harnessed somehow.
Then you could say, all right, well, can you repeat that trick? Is there any other physical process that even with a quantum computer, it takes exponential resources to simulate it? If so, maybe there's some other kind of computation even beyond quantum computation that, in principle, could be even more powerful. So, you look around. You say, well, where could that be? One candidate was in quantum field theory. In principle, according to quantum field theory, you have infinitely many degrees of freedom even within a finite volume. So maybe that's kind of like having infinitely many qubits, which you might not be able to simulate efficiently with finitely many qubits. You have to check carefully. Maybe you can discretize things and approximate things and thereby simulate with a standard quantum computer it after all. But it's not completely obvious right off the bat. So, that's motivation number one. Motivation number two is that these quantum field theory processes are important in particle accelerators. They're important in nuclear physics, and so on. Sometimes, they're actually quite hard to simulate, so it's, in a sense, a practical problem to simulate them. You've got the theory. You've got the experiment. But you need to calculate somehow the consequences of the theory to get the predictions to compare to experiment. And sometimes that's just computationally too hard. So, in such cases, you can't carry out the scientific method because of computational limitations. I had much better reasons, rational reasons, for pursuing that research project, whereas on the stuff about link invariants was motivated by more personal and mystical reasons. Unsurprisingly, the papers that I published about the field theory side of things were much more successful and highly cited.
ZIERLER: Stephen, the identity crisis that you talked about at MIT where the physicist said that this belonged with the computer scientists, and vice versa, did that not exist at Caltech simply because IQI was a place where those academic distinctions either didn't matter or where those specialists came together?
JORDAN: I don't know. I think it might've been just that John Preskill was really good at raising funds. He could manage to build a group as big as he wanted, single-handedly. [laugh]
ZIERLER: What was the research culture like at IQI? Where was there competition, where was there collaboration among the postdocs?
JORDAN: I would say it was very collaborative. Basically, there were two kinds of relationships you would see. One is where you each work on fairly distant subjects, and you listen to each other's seminars, and you have a certain amount of mutual respect but you don't really engage on joint projects because your specialties are too far apart. That's one kind. The other kind is you have shared interests, and you join forces and work together and write co-authored papers. I didn't really see competition much at all.
ZIERLER: Among the shared interests, who were some of the key people who had the same interests as you, and in what ways did you collaborate?
JORDAN: Well, Robert König knew a lot more about topological invariants than I did, and the deep mathematics behind them. I had a certain conjecture that I was trying to prove in a laborious manner using brute-force proof methods. I ended up collaborating with Robert König, and also with Ben Reichardt, to prove the conjecture the right way using the tools that are really most suited for it. That was one example of a collaboration. On the field theory stuff, I collaborated directly with John Preskill as well as with Keith Lee, who I had known at MIT when we were both there, and moved to Caltech shortly before I did. Keith was in high-energy physics. So, the three of us worked together on that, which was quite rewarding. John runs a big group, so not every graduate student or postdoc directly co-authors a paper with John, and works really directly with him on a project. But I got to do that. We didn't quite finish it while I was still there. After I had moved to NIST we continued our collaboration remotely, and finished it, and published it.
Even now, I'm still working on some follow-up papers on that same topic. We are now improving upon the quantum algorithms that we initiated in our 2012 paper. Back then, no one was working on quantum algorithms for simulating quantum field theories. Ours was not, strictly speaking, the first paper on the topic of quantum algorithms for simulating quantum field theories but it was really the first one that considered the problem in a thorough way from end to end. So we were kind of initiating a new line of research. Now, there are lots of people working on this subject because, for one thing, Department of Energy has allocated funds for quantum information initiatives, and so a lot of the Department of Energy-funded scientists in high-energy physics, nuclear physics, plasma physics, etc. now want to do cross-disciplinary things that combine their traditional expertise plus quantum computing. So, there's a lot of work now going on at the intersection of nuclear and particle physics and quantum field theory and quantum computing. But back when I was at Caltech, this was not the case at all. It grew into an established subject with its own conferences and workshops, and its own body of literature in which people build on each other's ideas. There's now a little community around it. That's been very nice to watch.
ZIERLER: Stephen, among the research areas that were very distinct among the scholars at IQI, is that to say that by the time you were a postdoc at Caltech, quantum information had matured and spread out sufficiently that you could be working on something that was really totally different than somebody who might've been an office mate to you?
JORDAN: Yeah. Well, people tended to specialize. It was not humanly impossible to be a serious researcher in all of those areas. But, in practice, for most people, it was a stretch, and so you tended to specialize. There were certain people who specialized more on the quantum communications and information theory side of things; maybe also cryptography. Then there were the people like me who focused on quantum algorithms, quantum complexity, and so on. A third type of people were focused on physical implementations of quantum technologies. It was rare to find someone who was really an expert in all three areas.
ZIERLER: Now, you were there in the years as the run-up to the creation of IQIM as an NSF site. But I wonder, to foreshadow those developments, if there was greater integration among the condensed matter theorists and the quantum information theorists that might've been the point to which John Preskill said, "We need to formalize this into a bigger center."
JORDAN: There were definitely people who dropped by from time to time from condensed matter theory. But I would say it seemed clear that there was potential for a deeper and more regular engagement, which I think indeed has picked up since the creation of IQIM. [pause] In some ways, one of the bigger successes in quantum information is that concepts and theoretical techniques from quantum information have been fruitful in condensed matter physics over the past decade or so. It happened in condensed matter physics, I would say, first. Then a similar phenomenon started to happen in high-energy physics. I think actually John was usually a step ahead. He picked up on these things very early or even foresaw them. He and his institution and his graduate students and postdocs benefitted from that element of foresight or good instincts or whatever it was.
ZIERLER: Was there a general optimism during your time at Caltech that the development of quantum computing was, first of all, something that everybody in one way or another was working toward, and that it was something achievable in the not-too-distant future, however you might define that?
JORDAN: I think the experimental progress was going pretty well. It was clear that it's a long road. But there was no sense of being stuck. There was a sense that we have this 100-mile long uphill road. But we're going along at a decent clip, and so I think there was a fairly positive sense about that. Certainly, it's also the case that some of these theorists didn't and probably still don't care whether actual quantum computers are ever built. The fact that they are, in principle, possible is enough to make them interesting mathematically, and then it's just a mathematical subject to study. That's not actually such a rare [laugh] viewpoint.
ZIERLER: Sure. What do you see as your own contribution in that metaphor of keeping up progress on this 100-mile uphill?
JORDAN: My focus, until recently, was almost exclusively two things. One is quantum algorithms. That is, what can we use these quantum computers for once we have them? This is partly a practical question about what value will they have to humanity as a technology, and partly a foundational computer science question about what's the fundamental power of this model of computation, and how does it compare to other models of computation such as polynomial time classical computation. My focus has been on the algorithms and complexity theory side of things, except that in the past maybe five years or so, a big part of my focus has been on applied research around quantum-inspired algorithms.
ZIERLER: Was this an area of research that NIST was interested in? Is that the point of connection as your next opportunity after Caltech?
JORDAN: Well, NIST, I think, had two reasons for being interested in quantum information and quantum computing. One is that they're part of the Department of Commerce, and they're responsible for setting cryptographic standards that get used in the non-defense sector. For example, when you buy something online with your credit card, the transaction is secured using standards that are set by NIST. Now NIST needs to set new ones that are going to be secure against quantum attack. The second reason is that NIST is thought of as the national metrology laboratory of the United States. Many countries have national metrology laboratories. Their general goal originally, back in the 19th century when these things were mandated, was: can you weigh bushels of grain accurately to make sure that you're not getting ripped off in international trade and things like that. But, eventually, when you're measuring time with atomic clocks and so on, the applications are more esoteric. Atomic clocks get used in GPS satellites, or perhaps to certify that fraud is not being committed in high-frequency trading on Wall Street. But it all evolved from those original weights and measures institutes. A lot of the best metrology tools come from atomic physics and involve quantum techniques that are very closely related to the experimental techniques that you would use to build both superconducting qubits and trapped-ion qubits. For both the cryptographic standards reasons and the metrology reasons, it was a natural thing for NIST to have a presence in quantum computing, and they hired into that. As far as the specific things that I worked on, I was really given free rein, which was very nice. My division chief was Ron Boisvert, and he created a very sheltered and very nice working environment where I could just pursue my research directions wherever they led.
ZIERLER: Now, you took the job at NIST before the appointment with Maryland, if I understand correctly.
JORDAN: Yes, I bought a house right next to the NIST campus because I liked having a short commute, not knowing that a couple of years down the road, I would be commuting 45 minutes east a lot of the time—
ZIERLER: [laugh]
JORDAN: —to go to University of Maryland [laugh] in rush hour. [laugh]
ZIERLER: This does suggest though that at the beginning, NIST provided you with all of the academia, as it were, that you needed at that point where an academic affiliation at a university did not seem necessary to you from the beginning.
JORDAN: Yeah, that's right. The thing that got added to the mix once I picked up the University of Maryland affiliation is that I started advising PhD students. Maybe actually that started a little bit earlier in practice. I was sort of de facto research advisor of one student, I think, slightly before things became official. My first student was Michael Jarret, and he was working in quantum foundations. He wanted to make a switch to something a little bit more technical rather than quite so philosophical. So, he started working with me, and I ended up becoming his PhD advisor eventually.
ZIERLER: In what ways did the affiliation at Maryland influence or not what you had already been doing at NIST?
JORDAN: It influenced my working style because at NIST, I was not running a research group. I would drive in in the morning. I'd go to my office. It was very peaceful and undisturbed. Sometimes, I would close the door, and I would pull out my books and study things, and try to prove theorems, and calculate stuff. Sometimes, I would talk to people such as my office mate, Yi-Kai Liu, who I sometimes collaborated with, or I would talk via video conference or telephone with my remote collaborators, and that was it. For most people, including me at present, time is quite fragmented. People have calendars with back to back meetings, and they have to respond to 100 emails a day, and all this kind of stuff. It was really the opposite kind of scenario at NIST. I really was like Isaac Newton during the plague. He just kind of shuts the door, and he could concentrate. At NIST I had that kind of environment. Then when I got more deeply involved in the University of Maryland, it was a little bit of a different feel. I was having regular meetings to advise students, and I was attending more seminars, and I wasn't cloistering myself quite as much. Of course, it breaks your concentration a little bit compared to just sitting silently for nine hours in front of a calculation. But, on the other hand, it's a vibrant environment, and stimulating. It taught me a little bit about advising people on their research efforts, which is one of the primary skills for my current role at Microsoft as a research manager.
ZIERLER: Stephen, when you got involved at Maryland, what aspects of quantum science were well-developed, and what aspects were you really part of building up from scratch?
JORDAN: Well, building something up from scratch is a very ambitious thing, and I think the only time I really have achieved any claim to that was in helping to kick off the subject of quantum algorithms for simulating quantum field theories. I do think that among the quantum computing work that I've done, that's probably what I take the most pride in.
ZIERLER: In your interactions with the students at Maryland, to what extent did their interests really indicate ongoing forward progress in that metaphoric uphill climb?
JORDAN: I got involved with different things with different students. With Michael Jarret, mostly the focus was around optimization using adiabatic algorithms, and around the fundamentals of how powerful is the adiabatic model and what properties of the optimization problem influence whether the algorithm will work well or not. His natural talent was around technical mathematics, and so we proved a lot of theorems to lay some foundations in that subject. Then, for the most part, my other students were involved primarily in continuations of the work on quantum algorithms for simulating quantum field theories. There's a basic question of whether quantum computers can simulate quantum field theories in polynomial time versus do quantum field theory processes have a fundamentally different computational complexity than standard quantum computing. I think that's largely resolved; quantum computers can in principle simulate quantum field theories in polynomial time. But then there's a more detailed question about how do you actually do it in a reasonably efficient way? That's a subject where you make incremental progress. A lot of the other work with students was on making incremental progress to get better and better quantum algorithms for simulating quantum field theories. That was pursued with Aniruddha Bapat, Troy Sewell, and Ali Moosavian.
ZIERLER: Now, the work you were doing at NIST, as you described earlier, there is a connection between NIST's mission and quantum science. Was any of your work specifically responsive to that mission, or you were purely off in your world, happy to be able to do your own thing?
JORDAN: I did serve on the working group on cryptographic standards for achieving security against quantum attack. That was not just following my nose and doing my own research. That was sort of a real job, which took up a fraction of my time at NIST. Also, while I was there, I got invited to participate in some work at the Office of Science and Technology Policy, in which we were drafting documents to formulate what eventually became the National Quantum Initiative. So, that was my other little adventure into more traditional government activities as opposed to being a very free-spirited and pure research scientist who happens to be employed by a government lab.
ZIERLER: Tell me about the circumstances of joining Microsoft.
JORDAN: In 2018, I became interested in the private sector. At first, I looked into working for startups in quantum computing. I was pretty concerned about the wellbeing of my students because I'd built up a research group at University of Maryland, and recruited students and postdocs into it, and I thought I might be letting them down if I moved away. So, I was looking at startups which might let me remain in place, and work for them remotely, which was not so common back then. But, at the same time, I wanted to explore options. I also looked at the big technology industry players, and explored my options, got a collection of offers, and ultimately decided to come to Microsoft.
ZIERLER: Now, the idea that you could pursue what you wanted to at NIST, what assurances were you looking for or were you interested in having when you came over to Microsoft, or was that really not in the cards?
JORDAN: I was kind of open to changing things up a little bit at that point. I had been pursuing various threads that I had started as a graduate student around different models of computation, the power of adiabatic computation, quantum field theories, and topological invariants for a pretty long time. So, I was a bit open to considering other styles of work, or a shift in focus. I did ask some questions to get a sense of the research freedom at Microsoft. But in the end I took a little bit of a leap. I didn't know exactly what things would be like at Microsoft. In fact, what I was planning to do and what I ended up doing were fairly different, I would say.
ZIERLER: What was your first project?
JORDAN: At first I had two projects. One was continuing a research project that I had initiated previously at NIST / University of Maryland about simulating wave propagation using quantum computers. That was something I was still working on and finishing when I arrived at Microsoft, and that was continuing in the same style of academic publication-focused work that I had previously been doing since 2003 or 2004. The other project was a collaboration with a group at Case Western Reserve University, which develops magnetic resonance imaging technology, which ultimately is sort of a quantum technology. You're manipulating the nuclear spins in people's bodies to do the imaging, and it's not such an unusual thing for physicists to conduct work on this topic but it's not something that I ever anticipated. The idea of our project was to use our quantum-inspired optimization methods for automated design of pulse sequences to run on MRI machines and make them work better. I volunteered to work on that, and that became my primary focus for a long time. It was very hard, and it took about three years.
ZIERLER: What was so hard about it?
JORDAN: Basically, it's ultimately experiment-driven. You have these machines, and they cost like $1 million. You would think that they're very well characterized, and everything is known about their systematic errors and all their nonlinearities and quirks. But, actually, it's not that well characterized, and so you really have to run a lot of experiments before you can make a good model of how these machines behave. It's only once you have a good model that you can start to use that model inside your computer simulations and your optimization processes. That was actually the hardest part. Getting the optimizers to run well, we had expertise in, and we got that running fairly quickly. But the really hard part was making sure that we were optimizing the right thing. In optimization we call it the cost function. We needed to work very hard to reach the point that when computer produced an optimized plan for how to use these MRI machines, which was predicted to work well by our cost function, it would actually work well when we tried it on an actual machine with an actual person in it. It took a long time before things that were predicted to be good actually were good when we tried them. That was the hardest part.
ZIERLER: Was it fun operating in a more applied environment?
JORDAN: Yes, it was. It affected me— especially because it was medical. There were people who had come in with brain tumors, and they would have to lie in these machines, and you can run these scans on them. With optimized pulse sequences, you might be able to see tumors that you would miss otherwise, and someone would live who would otherwise die. Or, if you could make the scan faster, you could get more throughput, and the cost would be cut in half, and more people would have access to it. People also have to hold still in these machines. Little kids can't do it because they're too wiggly, and so sometimes it's necessary to give the kids sedatives, or even knock them out with anesthetics. But if you can do a really fast scan, then maybe a kid could actually hold still long enough, and you wouldn't have to drug them. It really hits you hard. It hit me hard anyway. I had never experienced anything like that before. It was very motivating.
ZIERLER: Besides operating in such a different environment, what aspects of your education or research were useful to succeed in this new endeavor?
JORDAN: I had to do a lot of computer programming. I had played around with computer programming as a kid, starting in elementary school, as many people do. Programming in a more serious way, especially for scientific computing, was a skill I learned during my undergraduate research studying graphene with Vincent Crespi at Penn State. When this MRI project came along it was clear that it would involve writing some code to simulate the MRI machines. I said: "Yeah, I can do that. I'll just dust off my old skills that I have from 2001 or whatever." [laugh] That's, to a large degree, what I drew upon. The other import skills were general mathematical knowledge such as understanding things about Fourier transforms and having general scientific instincts about not engaging in too much wishful thinking, not jumping to conclusions too preliminarily based on weak evidence, and all this kind of stuff, which really doesn't come naturally to humans. It only gets beaten into you by painful experience of using your natural instincts and wishful thinking and premature conclusion jumping and wasting months of your life because of these mistakes. You eventually get trained out of it. [laugh]
ZIERLER: Stephen, just to bring the conversation closer to the present, what have you been involved in? What have been some of the major research projects over the past two or three years?
JORDAN: The MRI work was the biggest one that I worked on directly. Currently, I am leading a team to adapt these same kinds of algorithms that we used for designing MRI scans also to other, more widespread, industrial problems. There are many problems where you have some really large-scale process about, say, the power grid, or transport networks, shipping, or the operation of some chemical plant, or the operation of a data center. One way you can improve the efficiency of those things is, replacing some component. For example, maybe you use a new material for the wires that will have slightly lower transmission losses. Sometimes this is worth it because maybe you improve efficiency by only a percent, but because it's operating on such mind-boggling scale, a percent means hundreds of millions or billions of dollars a year. But a different way you could improve efficiency is without any new components, just coordinating things in a slightly smarter way, routing things differently, or allocating things differently, or scheduling things differently. That's purely a software problem. You don't need to retrofit any hardware. So, that's just waiting for someone to come up with better algorithms, and you just press a button and, all of a sudden, you're saving these resources. Maybe you're saving people's time. Maybe you're improving energy efficiency and preventing unnecessary carbon dioxide emissions. Energy and people's time both cost money so, of course, you're also improving the bottom line of whatever business you're applying these optimization methods to. So, it's also an inspiring subject. It's not quite as visceral as with medical things where you see individual patients and so on. But if you use your imagination, and you think about the numbers involved and the scale of the impact, then it can also be very inspiring. So, that's my focus now.
ZIERLER: Stephen, now that we've worked up to the present, a few broadly retrospective questions for the last part of our talk. Is there something that you learned at Caltech—an approach, a sense of collaboration, a way of looking at the science—that has stayed with you, that informs your work, and presumably with new projects that you might not see coming in the future, will always be present as you approach these things?
JORDAN: Yes. Well, I think I learned from seeing examples at Caltech of being really knowledgeable. Like many physicists, I was quite influenced by Richard Feynman, and read various books of his, some of which advocate rediscovering things for yourself, because it deepens your understanding. I think that's absolutely right. But you have to be careful not to misinterpret it and take it as a license to be lazy about learning from the textbooks and from the journals. At IQIM, there was a culture of really knowing the literature, really knowing the prior work, really knowing the textbooks, and so on. So, I think that was a positive influence on my attitude. The main big-picture lesson from those years would probably be that. Whereas the main big-picture thing from the MIT years I think would be to really watch out for wishful thinking, and really try to do the "fail fast" thing. Suppose you have some conjecture. Don't start trying to prove it. Start by trying to look for counterexamples or reasons why you think it's wrong. Otherwise, you can waste time on dead ends, and you have to be very careful about that. I'd say Peter Shor and Scott Aaronson were both good influences who said that, before you try to prove something with rigorous mathematics, you should already be pretty much sure that it's true. I think I do have some fairly specific lessons from my life.
ZIERLER: As quantum computing is getting closer and closer to scalability, what are the benchmarks that you'll be looking for? From your area of expertise, from the things you've worked on, what unique vantage point might you have to look at ways in which quantum computing will become more and more feasible, more realizable?
JORDAN: I think the big impacts are going to come with error-corrected quantum computers. Right now, there are a lot of interesting things being done with what are called NISQ machines, a term coined by John Preskill, which are noisy quantum computers. But actually having impact for real problems that are of economic importance that you can't already solve with state-of-the-art methods on serious classical supercomputers is a high bar. Over the past few years, I've spent hundreds of hours talking to people to learn exactly where that bar is in different domains. My conclusion is that you probably need error correction to get there. So, I think the key milestone that I'm waiting for is for someone to do a two-qubit gate on logical qubits fault tolerantly with error correction. That will be when I pop the cork on my champagne. [laugh]
ZIERLER: Now, is that—what is that? Is that a theoretical breakthrough? Is that an experimental breakthrough? Is it a eureka moment?
JORDAN: That would be an experimental breakthrough. I think the way we get there is through a very substantial engineering effort that moves towards it gradually, and eventually crosses that milestone.
ZIERLER: Is that to say that, right now, the theorists are out ahead of the experimentalists?
JORDAN: Yeah, in a way. It's possible that a theoretical breakthrough could hasten this milestone. Specifically, if someone comes up with some new class of quantum error-correcting codes and fault tolerance schemes, it could help. The current frontrunner is surface codes and magic state distillation. The surface codes were discovered by Alexei Kitaev quite a long time ago. There's a lot of work that goes on in quantum error-correction, and it has many different goals. But one big one has always been to outperform that scheme, and no one's achieved it yet. Maybe surface codes plus magic state distillation is just the best that can be done. Maybe a better scheme doesn't exist in the mathematical space of possible schemes. But maybe it does. It's hard to say. So, if someone discovers something like that, then it would be a theoretical breakthrough that would change the timelines of when we reach that experimental milestone.
ZIERLER: Finally, Stephen, for you, last question, what's next on your agenda? What are you excited about in the future for your own work?
JORDAN: What I really am focused on now is getting results from research transitioned into practice. We've now got some successful examples. We've released some information publicly about our collaboration with Azure Storage, which is a group at Microsoft that oversees data storage in our gigantic data centers. We've worked with them, and optimized their processes, and made very big improvements, and big savings. So, that was an exciting development. I'd like to see if we can repeat that with some other process, and also see if we can similarly empower some customers. Whether it's from power utilities, biomedical, logistics and shipping, whatever it is, I would like to achieve large economic impact. Then you could point to it and say, even to a cold, hardheaded bean-counter, that all the investments in having people like me sit in their chairs, and use up computer time, and collect their salaries, have paid off many, many times over from these applications. I think with the Azure Storage thing, we have achieved it already. But I would like to achieve it with more examples. There are many different kinds of goals you can have, and you don't have to have only one. But this is one that I am currently excited about.
ZIERLER: Well, Stephen, it's been a lot of fun talking to you. I'm so glad we were able to do this. I'd like to thank you so much.
JORDAN: Yes, thanks a lot.
[END]