Gary McGraw: This is a Silver Bullet Security Podcast with Gary McGraw. I’m your host, Gary McGraw, vice president of security technology at Synopsys and author of “Software Security.” This podcast series is co-sponsored by Synopsys and IEEE Security & Privacy magazine. For more, see www.computer.org/security and www.synopsys.com/silverbullet. This is the 150th in a series of interviews with security gurus. That’s 150 months in a row, and I am super pleased to have today with me Filippo Valsorda. Hi, Filippo.
Filippo Valsorda: Hi, Gary. Very happy to be here.
Gary: That’s great. Filippo Valsorda is a cryptographic engineer building and breaking systems in Go. He works at Google on the Go Open Source project, where he owns the Go cryptography standard libraries. Previously, at Cloudflare, he worked on TLS 1.3 and DNSSEC. Filippo is best known for his 2014 Heartbleed vulnerability test. (Full disclosure: Synopsys SIG was involved in discovering Heartbleed.) Filippo grew up in Milan, where he earned a scientific high school degree in 2013. In 2009 he participated in the World Math Games Championship as part of the Italian team, and in 2013 he won a bronze medal in the Italian Mathematical Olympiad. Filippo currently lives in New York City.
So thanks for joining us today.
Filippo: Yep. Thank you for having me on.
Gary: Regular listeners know that I’m passionate about programming languages, just like you. Although I think you know a little more than I do. In fact, my start in security was driven by Java’s introduction in 1995. You’re super active in the Go community, obviously. So what is Go like as a programming language, and why do you like it?
Filippo: Well, something I always say to people is that the thing that drew me to Go is that it has a very low mental overhead for programmers. I feel like even when I’m tired in the evening or maybe I’m having an off day, I can still get code done in Go, because it gets out of your way and it lets you focus on what you’re trying to build.
Gary: That’s cool. So just to be clear for some people who haven’t used it or come across it, it’s compiled, statically typed, with memory safety. It has garbage collection. So it sort of combines some properties of dynamic languages and some properties of compiled languages, I suppose.
Filippo: Indeed. I think that most of the original team expected it to be a replacement for C, and over the years a lot of good new Go developers came from dynamic languages like Python and Ruby. And this was, I heard, initially surprising to the team. That is a good testament to how easy it is to use and how low its static typing system overhead is.
Gary: Right, so you just sort of program like normal, and all of that stuff, garbage collection and memory safety, happens behind the scenes and pretty efficiently.
Gary: So in your view, what role do programming languages play in software security and in security in general?
Filippo: Well, we’re all familiar with the endless list of vulnerabilities that are caused by lack of memory safety in languages. So these days, that’s pretty much table stakes for any security application that does not have strict requirements for being in C. So definitely memory safety is the one big advantage of programming in Go, from a security point of view. But beyond that and more into my realm, Go really tries very hard to provide safe APIs by default, including for cryptographic operations.
Gary: Yeah. That makes sense. So when you’re doing cryptographic operations, you feel confident that the memory management stuff is not going to screw up and you’re going to leak things like key information.
Filippo: Indeed. Heartbleed could not have happened in a memory-safe language, because there was just no way to overwrite buffers, right?
Gary: Right. Right. Interesting. So let’s contrast this with, say, your view on dynamic languages in security. Maybe something like Python.
Filippo: I have a past as a Python programmer myself, and there are definitely contexts in which Python is a perfectly fine language to work with security-sensitive data. The two things that are hard to do with Python in security-sensitive contexts are constant-time code. Since you have much less control over what the program is doing, what the actual processor is doing, it’s harder to make sure that all operations are constant time. It’s hard to know how exactly the timing profile of your code is going to be. And of course, sometimes Python just has too much runtime overhead. It isn’t fast enough for the application, and that’s the case in which I think Go shines on the server side.
Gary: Yeah, I think the timing stuff is interesting because there are a bunch of side-channel attacks that take advantage of timing in different situations to do stuff like expose keys.
Filippo: Yeah, exactly. And yeah, those are very relevant in cryptography. The coverage of constant-time coding Go is partial, but most of the widely used cryptography algorithms are constant time and implemented in the standard library. Also, the other thing that you get out of any statically typed language is that you have a higher level of confidence about any big codebase and refactors, that you’re not introducing logic bugs, for example, just by mistake, because maybe something that was not supposed to be a certain type is a string or something like that.
Gary: Exactly. So it helps you catch mistakes. Whereas a non–strongly typed language, you might have the mistake go by and then some weird error occurs.
Filippo: Exactly. Yeah, the focus of Go is to make it easy for programmers to be productive, and part of being productive is not making mistakes or catching mistakes easily. And that, of course, has a big security impact as well.
Gary: Yep. So you’ve done tons of applied cryptography work. What got you started in applied cryptography?
Filippo: I wish I had a better answer for this question, because it comes up often, but the first things I remember are just doing the Cryptopals, which are this set of cryptographic challenges that teach you to build a cryptosystem with a defect, with some mistake, and then teach you how to break it and how you could break it in the real world. They start kind of easy, and they get all the way up to, “Hey, read this paper. So this paper actually broke SSL in 2006. Implement it.” And I found those extremely fun, and I think that was the tipping point where I realized that this was something I was into.
Gary: Cool. When was that?
Filippo: This would have been 2012, 2013.
Gary: Wow. Wow. So what advice do you have for somebody just getting started with cryptography? Do you think that doing a set of exercises like that is the best way to go?
Filippo: Yeah. Of course I’m skewed on the cryptography engineering side. I know little about how you become a cryptographer in the academic sense. The ones that design, you know, the primitives.
Filippo: The ones that make the AES and SHA candidates of the world. But in terms of cryptography engineering, breaking real cryptography and writing a cryptographic code is where I would start.
Gary: You know what’s funny, Filippo, is that sometimes cryptographic mathematicians that come up with those primitives are not really very good at coding. So they come up with a great set of primitives, but they can’t implement them without mistakes.
Filippo: Yep. That does happen and sometimes you read these papers, which are brilliant, but the actual “How do you implement this?” is hidden so deep in mathematical notation that it takes two days to reverse engineer the paper before you can implement the code.
Gary: I know that problem. In general, do you think that cryptographic implementations are getting better or worse over the years you’ve been doing this?
Filippo: They are definitely getting better. The amount of scrutiny that came post-Heartbleed has improved the ecosystem, and we are now focusing a lot, all the attention it deserves, to the foundation, the libraries that we are using. While before Heartbleed, in the early 2000s and late ’90s, it was a very much overlooked part of the foundation. And then Go had the advantage of starting really recently. And most of the early code was written by Adam Langley, and he did an excellent job, so I’m very happy about the code that I now maintain.
Gary: Cool. So what’s your favorite book on cryptography?
Filippo: Oh favorite book. Interesting. So a recent book by J.P. Aumasson, two-word title, “cryptography” in the title. I’m just blanking on one of them.
Gary: That’s OK. We’ll find it, and we’ll put it into the links for the show. [The book is Serious Cryptography.—Ed.]
Filippo: Sounds good. The good thing about it is that it’s an approach to learning cryptography that is modern, while a lot of the classics are very interesting if you have an interest in how cryptographic engineering has developed over the years. But these days they teach you modes of operations that you would not actually use in practice, because safer primitives have been developed in the meantime. And when I say this, I don’t mean the old ones are broken, but they are particularly easy to implement wrong. And that was not something that anybody paid real attention to until 5, 10 years ago.
Gary: Right, right. That makes perfect sense to me. So what do you think was or is the biggest challenge in creating and curating the Go cryptography libraries?
Filippo: So it’s definitely…the hardest thing there is deciding what to reject and then designing the APIs for the things that do get included. The hard part, you might not believe it, is not actually implementing the cryptographic primitives—which is, I find, the most fun part—but deciding what should be exposed to what API to make it approachable by developers so that there are no “footguns” just laying around in the standard library, just waiting for people to shoot their feet off with.
Gary: So you’ve done a lot of breaking or attacking cryptography. Where did you learn your approach to doing that?
Filippo: We’re definitely going back to the Cryptopals there. They’re a very good approach of presenting you a broken cryptosystem and giving you just enough nudges to let you build your own way of breaking things. How I like to think about this is that you essentially develop capabilities. For example, when you’re doing an adaptive attack, you build an attack that reveals one bit of the key. And that doesn’t get you anything, right? Because there are 256 bits in that key or maybe more. But once you have the capability to reveal one bit, if you can build a capability that lets you figure out the next bit based on the previous one, you put together these two capabilities, and you broke the algorithm completely.
Gary: Right, Right.
Filippo: And sometimes they do very weird things that have nothing to do with the previous bit. You just have to figure them out as their own self-contained capabilities and then put them together like LEGO pieces to build an attack.
Gary: Yeah, we always used to say it was kind of like mountain climbing. You have a foothold here and a handhold here and ways to do it, and you combine all those things to build an exploit.
Filippo: Yep, exactly. That’s also why I am always worried when I see an attack that stops just short of doing anything practical. Because I know that that’s a collection of capabilities that shouldn’t be there in a safe system, and it’s just waiting for someone to find a capability to do something that will combine with that and build a full attack.
Gary: Right, right. So that’s why you have to take each vulnerability slightly more seriously than on its own.
Gary: We’ll be right back after this message.
If you like what you’re hearing on Silver Bullet, make sure to check out my other projects on garymcgraw.com. There, you can find writings, videos, and even original music.
So you worked on TLS in a past life, when you were at Cloudflare. I just kind of wonder, do you interact with Adrienne Porter Felt on the Chrome team at all?
Filippo: Not directly. I follow her work, and I’m very impressed by how she’s able to make this dry code of ours have the correct and good impact on the world as it should have. So I’m a huge fan of her work, and I did listen to the podcast episode with her on the show.
Gary: Cool. That was your prep?
Filippo: Yes. Yes, it was.
Gary: Very clever. You’re a clever person. So you appear to be mostly self-taught and still working in high geekdom at the highest level. So tell us how that happened and whether you have any plans for more formal schooling or anything like that.
Filippo: So there is a way to tell the story that is, like, “Oh, I knew that university wasn’t for me. I knew what I wanted to do, and I went and did that.” That would be lying. That’s not what happened. What happened is that I was in Italy, and I was dissatisfied with the computer science programs that were offered in Italy. My girlfriend at the time was enrolled, and I wasn’t really liking the program as she was going through it. And it was too late to enroll into university abroad, so I essentially just procrastinated my way into skipping the first year.
Gary: I love it.
Filippo: By the time the second year came, I had attended the Recurse Center, which is this amazing three-month program in New York. It’s free and you get to just work on your own open source project together with other people that are there for the same reasons. It doesn’t have lessons, but it’s definitely one of the best experiences I’ve had as a programmer. Yeah. By the time the second year came around, the Heartbleed test had happened, which was definitely a lucky break. And I found myself at Cloudflare.
Gary: Yeah, so I guess your education plans were molded by academia in Italy. Which can be stodgy at times, I do recall.
Filippo: Yeah, it’s just far behind. Or at least it was five years ago. And it was, while I was already getting involved with Go and with misuse-resistant cryptography—I don’t think we were using that term at the time, but still the idea that we need to make things safe—and all these new modern pieces, and the programs were still very much stuck in the ’90s with writing C on paper. So I was not a fan of that. I had spent, though, middle school and high school learning on the internet, Cryptopals, the Coursera Cryptography 1 course by Dan Boneh.
Gary: Yeah, that’s a fantastic course. Dan’s a great guy.
Filippo: Yep. His book is also worth checking out. He has a public PDF of the book for his course, very recently updated. And it’s a big book, but it’s excellent for reference.
Gary: So you’ve contributed to lots of open source projects, including Go, DuckDuckGo Zero-Click plugin, homebrew, vanitygen, bitaddress.org, Axel, Archive Team, and so on. What’s your philosophy on open source versus proprietary code?
Filippo: Well, even now my full-time job is entirely open source, and I’m extremely happy about that. I owe most of my career to open source. I got started when the youtube-dl maintainers, which is this add-on tool to download videos from hundreds of different video websites, just entrusted me to become a maintainer, and I just spent a year working on that and learning a lot. So I’m definitely an example of how it’s an excellent way to grow as a programmer. And I tend to work and do security research almost entirely on open source software. So I don’t think I can really give you a comparison here, because I don’t really have a comparison.
Gary: Yeah, that makes sense. But for you, open source has been super fantastic. I do want to ask this, though: How much should people worry about the security of open source? I know that’s a silly question, but I’m going to ask it anyway.
Filippo: Yeah, I hear it from time to time. I’m not sure it’s a strongly relevant metric except in certain threat models. If your threat model involves not trusting the build system and the artifacts that you’re running, you will want open source software for the simple reason that you don’t trust anyone to do the build for you. So that’s one threat model in which you will want open source. But otherwise, I’ve definitely seen how open source works to get security review from the community, but I’m also not entirely sold on the idea that “with enough eyes, all bugs are shallow.”
Gary: Yeah, I totally disagree with that as well. I mean, it depends on the eyes.
Filippo: Yeah, exactly. You still need to get the eyes on your software, and while you might get some just by being open and being out there, it still needs either outreach or it still needs money to get people to work on it. I’m very grateful that Google pays me to work on the Go cryptographic standard libraries, and I think that’s an important coordination role, if nothing else, that wouldn’t work if there wasn’t a person with that role. Which I’m not just saying because it’s why I get paid.
Gary: No, I think that’s exactly right, and it’s insightful. I guess one of the other threat models that’s worth talking about is a lot of people use open source components without really keeping track of how they’re updated or who’s running the project, and they continue to use old ones that have been deprecated or that have known bugs. And that turns out to be a problem. And they think, “Oh, well, it’s got to be secure because it used to be secure.”
Filippo: Yeah, that’s definitely a problem. And to be fair, you can’t ask someone that is just putting their work out there for free to give the same maintenance promises that a company can provide. So there are pros and cons, and I think that open source is, in general, a positive, but it is not a full solution. I believe that models where the software is open source but you also have other reasons to believe that it’s maintained and secure is the best of both worlds.
Gary: Extremely well-stated. So I hate to ask this, but I have to: How do you feel about blockchain technology and cryptocurrencies?
Filippo: Oh, no. Why did you do this?
Gary: They made me ask it. I don’t know if you heard the episode with Nick Weaver, but he got a good chance to rant about how terrible that stuff is for 45 minutes.
Filippo: Let me think through my words here. Let’s start with the good things. It’s sponsoring some impressive cryptography work. Some of the zero-knowledge proof work that is state of the art is being done in the blockchain space. And it’s an interesting concept, and I’ve had fun with it myself. Five years ago now, I was breaking wallets because there were reusing ECDSA nonce values. So there’s definitely good engineering, good people, and good research in that space. However, the whole financial aspect is unsettling. Yes, it feels dangerously unregulated and free for all. But that’s all I’m going to say on the financial side. As for the technology itself, it’s kind of overused as a solution.
Gary: Yeah, well, people are just saying, “Let’s just throw a blockchain at it and see what happens.”
Filippo: Right. If the answer to, if you could ask yourself, “Would it be faster and still as good if I used Postgres instead?” and you have to think about it, you might not need a blockchain. And it’s going to be so much faster. I’m telling you.
Gary: All right. Let’s talk about speed and scaling a little bit. So in 2014, as you already mentioned, you built the famous Heartbleed test, and at times you were processing 25,000 checks per minute.
Filippo: Along those lines.
Gary: Holy cow. So how did you scale your work? I mean, I know Go is efficient and all, but didn’t you have to buy a pile of servers? How did all that work?
Filippo: I will say that I’m extremely grateful to the Amazon security team for sponsoring a grant of credits there.
Gary: Aha, so it was scaling through AWS.
Filippo: Yes, but also Go was actually surprisingly effective there. I started with a Python forking server, and that disintegrated extremely quickly. So I switched to the Go program, and I got it wrong, seriously wrong, for the first 24 hours, and it still ran. And that, by the way, is the first time Adam Langley, the author of the cryptographic library, heard my name. Because I sent him this, like, very wordy email saying how I was seeing packets that were arriving on one TLS connection come up on the other one, which sounds like a security vulnerability, right?
Gary: Yeah, it does. It sounds like a big giant leak.
Filippo: Yep, and he looked at my patch to implement the Heartbleed attack, and he was like, “You know you’re using a constant, right? A global, right?”
Gary: Yeah, and you were like, “Oops.”
Filippo: Yeah, I was using a global variable to just store packets. Ahhh.
Gary: Oh well. But you got it fixed, and then, like, so many people used that thing. We used it here. I remember we had to check our servers, and we were like, “All right, let’s just go use Filippo’s tool.”
Filippo: Very glad that it helped.
Gary: It was extremely useful. So why did you do it? What possessed you to build that test?
Filippo: So again, Adam Langley had built this tool for goto fail. Do you remember that? It was this Apple bug where there were two goto’s, a duplicated line, and you could bypass so that you’d get validation to it. And he built a testing page. It was just a page with a single string of text that would load only if you were affected by the vulnerability. And I found that a neat idea, like such an easy way to test if something is vulnerable, which usually instead involves running some script that might not run in your environment and that might be complex to obtain and compile. And I just found that a neat idea. I put together just a bootstrap page and expected it to be used by a solid 50 people, and that would have been fun. It was a late evening in Milan. I was just coming back from having drinks, and I was just, “Yeah, sure.”
Gary: You know, after you have drinks, you use global variables.
Filippo: Yeah, exactly. I slept, I think, eight hours in the following week.
Gary: Once it caught on. How many people used it? Do you know? Any idea, any wild guess?
Filippo: Well, people are hard to count, and I was trying very hard to keep the least amount of analytics as possible. But I know it did a number of millions of Heartbleed tests over a couple weeks, I think.
Gary: Fantastic, well, thanks for doing that. I love hands-on tests like that too.
Filippo: Yeah, and it was a 10-line patch. Actually, it’s what got me definitely into Go. I was looking around for a TLS stack that I could patch to do Heartbleed, and I spent half an hour looking at OpenSSL and just being like, “How does any of this work?” It was before they improved the codebase. And then I looked at the crypto/tls standard library package in Go, and it was a 10-line patch. You know, wrong patch the first time around, but it was extremely easy, very clean, and that got me into Go.
Gary: Fantastic. I love that story. OK, so now for something completely different for our last question. I just did some OK scuba diving in the Bahamas, but I’ve been diving all over the place, and I know that you like to scuba dive too. So what’s your favorite place to scuba dive?
Filippo: It was definitely in the Red Sea, a few years back, in Egypt. I am kind of sensitive to the temperature of the water, so that was my perfect diving spot. Beautiful colors, corals, fish, and lukewarm water that you could just dive into with the short suit. Which I don’t recommend anymore after a coral slashed my leg on a wall dive, but eh.
Gary: You know, that’s the price you pay for such beauty and tepid water.
Filippo: Yeah, exactly, and also yeah, I loved how quick it is to put on that little suit, and it was easy to move on to help others with the equipment. So being a bit arrogant, about “Sure, I can just do this with the short suit.” Nope, not anymore.
Gary: Well, thanks. This has been really fun. I appreciate your participating on Silver Bullet.
Filippo: Thank you very much for having me on. This was fun.
Gary: This has been a Silver Bullet Security Podcast with Gary McGraw. Silver Bullet is co-sponsored by Synopsys and IEEE Security & Privacy magazine and syndicated by Search Security. The July/August issue of IEEE S&P magazine is a special issue devoted to blockchain security and privacy. The issue highlights our interview with Nicholas Weaver from ICSI, who has a thing or two to say about cryptocurrencies and blockchain. Check it out. Show links, notes, and an online discussion can be found on the Silver Bullet webpage at www.synopsys.com/silverbullet. This is Gary McGraw.