Gary McGraw: This is a Silver Bullet Security Podcast with Gary McGraw. I’m your host Gary McGraw, CTO of Cigital and author of Software Security. This podcast series is co-sponsored by Cigital and IEEE Security and Privacy Magazine. This is the 117th in a series of interviews with security gurus and I’m super pleased to have today with me Jamie Butler. Hi Jamie.
Jamie Butler: Hi.
McGraw: Jamie Butler is the CTO and Chief Scientist at Endgame where he leads Endgame’s research on advanced threats, vulnerabilities, and attack patterns. He’s directed vulnerability research teams at a number of prominent companies including FireEye where he was Chief Architect, and Mandiant where he was Chief Researcher. Jamie has over 17 years of experience in operating system security. Jamie served as a Computer Scientist at the NSA and co-authored RootKits: Subverting the Windows Kernel, a book in my Addison-Wesley Software Security series. Jamie has an undergrad degree from James Madison University in Virginia, and an MS in Computer Science from University of Maryland: Baltimore County. He lives with his wife who he met at Black Hat 2005 in DuPont Circle, Washington, DC. So, thanks for joining us today Jamie.
Butler: Yeah, thanks for having me.
McGraw: You’ve been at Endgame just a short while, about nine months you said, working for Silver Bullet victim and good friend of mine Nate Fick. [both laugh]
Butler: Yeah, that’s correct.
McGraw: What do you do as CTO of Endgame?
Butler: So, I started back in March. I was charged with building out their research team. They had a lot of research in the past, but it wasn’t a clear business unit. We wanted to make sure that was on track with the company so we formed the R&D team here. We transferred a few people in internally that were already existing within the company and then we hired about 11 people.
McGraw: So you have a bunch of people reporting to you I guess.
Butler: Well, one senior director came over from an existing team and then I hired another director out of the NSA who is over our malware research and threat intelligence. I’m still looking for someone to lead up our data science effort.
McGraw: What are your responsibilities as CTO? It’s kind of a funny title; I know, I’ve had it for 20 years and people do different things when they have that title.
Butler: Yeah, I think every company is a little bit different with their CTOs. Some are more evangelists. Some are more—sometimes the CEO becomes the CTO after he founds the company and gets some traction. For me, the Chief Scientist part is really about the internal research and building detection into our product suite. But, the CTO is more the outward facing—talking to investors, talking to the press, and things like that.
McGraw: So, managing the brand and being the—I always call it—the hood ornament of the corporation.
Butler: Yeah. I mean, I still have real functions as the Chief Scientist. But yeah, it’s a lot of outward focusing things.
McGraw: What do you mean real functions? [laughs] Come on, Jamie. [laughs]
Butler: Well, you know. I’m a coder at heart. I could never get rid of my coding job to some degree. I like to take it with me wherever I go.
McGraw: Well, that makes sense.
Butler: Recently, I told the CEO, Nate, that I was writing some code for an end-point product and he kind of rolled his eyes and said “I’m not sure you should be doing that.” And I said “well, I’m not sure I should be either but it’s what keeps me happy.”
McGraw: Yeah. Oh, he’ll let you do it. He’s a good leader.
Butler: Yeah, he’s great.
McGraw: In its first iteration, Endgame was all about offense and, in my recollection, sort of played a role in this idea of active defense in the private sector. Has that changed?
Butler: I haven’t been here that long, but I can speak to what I know about the company’s past. They did start with an offensive mentality to help the government in various areas; particularly, it’s been well-publicized that in the vulnerability exploit type of thing. I don’t think they ever hacked back or the active defense if you consider it that way. Only at the federal level do they have the correct permissions or legal authorities to do something like that.
McGraw: It was all kind of bound up together with the CrowdStrike guys and there were a few start-ups that were talking up a big game in terms of this notion of attacking back. What are your thoughts about that by the way? Just as a philosophy, do you think that the private sector should be allowed to do that?
Butler: I don’t. I think it’s really difficult—the attribution piece of it. In the meantime, if you don’t get the attribution right, or maybe even if you get the attribution right, they’re using jump points or whatever else like you’re actually victimizing potentially innocent bystanders in the process. So, I don’t espouse the hack back and I think Nate’s pretty firm on that as well with our company.
McGraw: Yeah. I know he’s pretty firm on that himself, philosophically. One way of waging war is to get your enemy to attack your other enemy and watch them have a fight. And there’s no reason why that can’t happen in cyberspace as well. Like you, I believe we haven’t solved the attribution problem very well, and especially when it comes to short time periods. If you want to do something right now to whoever is attacking you, figuring out who that is sometimes takes months to figure out. So you believe we haven’t solved the attribution problem either.
Butler: Yeah, that’s correct.
McGraw: Of course, there are great offense ideas like rootkits and attack patterns that you’ve been working on for many years that play a really important role in security engineering too. Do you think that we’ve made progress in trying to adopt some of the thinking and ideas from the attacker community and use that to inform our engineering, or what’s your feeling about that?
Butler: I think we’re still a little bit early there. That’s what we espouse here at Endgame is really taking that offensive mentality—like how the attacker thinks and works and using that against them. I’m building kernel level capabilities within our product suite and so forth. So, I believe you need to have a very offensive mindset. We’ve hired a lot of people out of different organizations who have the authorities to do hacking and so forth. I believe we’re uniquely positioned to really deliver that capability to the marketplace. The only place I think maybe the industry is getting it kind of right is the exploit prevention side; there’s a lot of that going on right now. Even the larger companies like Microsoft and so forth are building it into their products.
McGraw: With like address-space randomization on the stack and that sort of thing.
McGraw: Those are good ideas and they are informed by attack patterns. But I’ll tell you what, you know we have this thing called the BSIMM, and the weakest of all the 12 practices that we keep track of is attack patterns. So I think we have more work to do to get people that do security engineering to really understand, and comprehend, and appreciate the attacker perspective.
Butler: Yeah. I was talking to someone the other day and I said “I come from an end-point world, operating system-centric, and I think we spent the last decade in building products that can create a lot of data. If you want to get data off an end-point there’s products out there that will do it, but they don’t have really any intelligence built into them, and so you have to have an expert really use those tools to make them useful. So, I believe that attacker’s patterns, and how they operate, and what their motivations are, and so forth, are important to build it into the product the capabilities to detect those things.
McGraw: I agree with that and the only way to know that is to study it and talk about it and write about it. You know that I wrote Exploiting Software a million years ago with Greg Hoglund who was your co-author of the rootkits book—that was a great book, by the way.
Butler: Thank you.
McGraw: We first wrote about rootkit stuff in chapter 9 of Exploiting Software, and we sort of figured we’d get a lot of flak for that; and we got less flak than we expected because I think that everybody understands that you need to know the attacker perspective in order to do what we’re all trying to do.
Butler: I had some of your very early books on Java security and so forth, and I had the Exploiting Software book before I ever met you through Greg. But when Greg and I put—we had to do a sample chapter and the introduction and outline for the publisher, Addison-Wesley. And we put it out there and they shop it around to some other experts in the field and the experts kind of give their thumbs up or thumbs down, “do you think this is a good idea?” Some—and if I said these people’s names you’d know who they were. I won’t do that, but they basically said “why in the world would you want to publish a book like this?” So, it was kind of tough at first getting the contract and so forth. A lot of people just want to—they think that you’re helping the enemy or whatever. I kind of come from the world where you had to do proof of concept in order for vendors to change their security.
McGraw: Yeah, so you have to show them what that means.
Butler: Like the FU rootkit that I wrote back in 2003. That was just to show basically Microsoft that this was possible to hide the processes in the kernel without anything wrong happening.
McGraw: Right. You can just be very very stealthy and they don’t even see you.
Butler: Yeah, and it took—I want to say it was like 4 or 5 years to patch, maybe even longer because basically their solution to it was in Windows. I forget what the first incarnation of PatchGuard was—if that was Windows 7-64.
McGraw: I think it was Windows 7. Yeah.
Butler: So, it was actually I think 7 years to patch. But, PatchGuard was the first thing that defeated it.
McGraw: Well, you remember that hooking system calls way early on in NT—heck, that worked before the FU Rootkit came out for many years as well. There was sort of a one byte hack to NT.
Butler: Yeah, people go look for patches for those system calls, and Greg and I released on TechTV back in the day. I think it was maybe 2004—no it must have been 2005—we released Vice, that would basically go around and catch the hookers. It would look for people patching your system call table and other things like that.
McGraw: You and Greg always had the cleverest little names for your stuff.
Butler: Yeah. But it was funny. We went on live television and gave this 6-minute presentation. And it was somewhat to promote the book and so forth, but we wanted to release something useful for the audience. And this was a very general audience; it wasn’t a security audience. We came back and there were like 83,000 downloads of the tool on rootkit.com that Greg founded and I helped administrate. It was crazy. We had people writing in like “hey these are the hooks I’m finding.” Microsoft actually hooks themselves.
McGraw: Yeah, I know. [laughs] Microsoft research built the whole detours thing with hooking.
Butler: Exactly. And now they sell the detours for 64-bit. And so, we had these mom and pops like emailing in to rootkit.com saying that they’re detecting these hooks and wanted to know if they were bad or good and it was a mess.
McGraw: Careful what you ask for. I remember when I did TechTV for the first time. I think it might even have been with Greg—or maybe it was with Viega before that. But, the amount of web traffic you get with television is astounding. I guess it has become more distributed, but TV is still a very powerful medium, indeed.
McGraw: So, tell us about operating system security. When you started with your rootkit work, at least there was a kernel. But it was the early days of kernel-land and things have changed a lot in operating system security, I think for the better. Do you agree?
Butler: They’ve definitely advanced a lot. I think actually Windows has advanced the most. I’m not a Linux expert by any means, but some of the old tricks that you do on Windows, you can still do on a lot of the Linux boxes. I’m really impressed by the amount of research and also the investment that Microsoft has put into their operating system. They’ve hired some great people in the security industry and things like that.
McGraw: Yep. Well, I think they really took software security seriously and integrated it, not just into operating system stuff, but also into their other products too.
McGraw: That’s good. That means we’re making some forward progress, I guess. Even though, sometimes when you look back, you wonder how little forward progress it is.
McGraw: I know you guys are using machine learning stuff at Endgame, and I don’t know if you know this, but I actually have a PhD in that nonsense from years and years ago. So, I’m really interested to know what kind of machine learning you’re using and what you’ve learned about that yourself.
Butler: I’m definitely not the expert in that. I can talk myself into a corner probably; but, we have two types of machine learning that we’re using. We’re using some malware classification right now on clustering and we’re using some supervised machine learning for that. We think potentially that we might enter some unsupervised machine learning on that because we want to try to trick our models and so forth, and get user feedback and see if it’s truly good or bad and—
McGraw: Are you using neural networks or are you using genetic algorithms or are you using symbolic machine learning? Do you know?
Butler: I know we’re not using genetic. I’m not sure about the other classes that you mentioned.
McGraw: I mean, supervised versus unsupervised is an important distinction. So, that gives us some idea. That’s something that’s pretty new. I bet that’s pretty fun to be learning about.
Butler: We have a good-sized team here with a lot of good experience. We have someone who we’ve hired out of national labs and things like that. Others that come from different computer security companies, and a few even from politics and things like that. So, it’s an interesting, diverse group of people.
McGraw: I think that we’re going to see a lot more machine learning in computer security because of speed issues and classification issues. The machine can do stuff faster, not always more accurately, but certainly faster than humans. And this notion that there are people sitting around in a SOCs watching the internet go by is the dumbest thing I’ve ever heard in my life.
Butler: [laughs] Yeah. In addition to some of the malware work that we’re doing, we’re also working on behavioral analysis of the host itself and how that host is related to other hosts in the environment and what’s going on there. Then we’re just starting to look at user behavior as well; kind of a pattern of life for that user and seeing deviations away from that.
McGraw: Yeah, there’s a big body of work about that. You have to watch out for this one little sneaky attack that involves changing the user profile really slowly over time. So you get into attack space and you carry out your attack and then you very slowly change your profile way back to normal which was actually seen in a national lab once. [laughs]
McGraw: That was trying to use anomaly detection stuff, but that’s all good. Are you still active on the rootkits.com thing with Greg or is that done?
Butler: That’s done. It was hacked a few years back.
McGraw: After the HPGary thing?
Butler: Yeah, with the HPGary thing, and so it hasn’t been put back online as far as I’m aware. I think that Greg still owns the domain and everything. We’ve chatted a few times about bringing it back up, but you know, its—
McGraw: Served its purpose.
Butler: Yeah, it has served its purpose. The time has passed. When we had it up, one of the problems we always had—and Greg was always frustrated by this and trying to figure out how to make it better—was just the amount of contributors was pretty low.
Butler: We had 80 or 90 thousand users. Now, not all of those were real users because we had no way to reset your password. So people would just create a new account. So anyway, we had a lot of users. There was literally like a handful—maybe two handfuls—of people who contributed on the site. That was always an issue getting new and fresh content.
McGraw: Listen, man, I saw that in the early days of static analysis tools for code review as well. We released this tool in 1999 and it got downloaded 15 thousand times in the first month, and then even more during the first year. We asked people to contribute rules to the rule set. Guess how many rules we got over the first year.
McGraw: Zero. [laughs] None. In fact, I had to force my own guys at gunpoint to make rules because it was such a pain. So I think this notion that everybody’s going to crowdsource security is pretty darn silly.
Butler: With rootkit.com, everything was open source. I had one person reach out to me—a very smart lady from the UK with a bug in the rootkit. It was actually blue-screening her system. She could point out where it was, but she couldn’t tell me what was causing it. It was still good feedback that there was a problem there, so that was nice. But really, there wasn’t a lot of other contribution that came out of it. We used to joke that probably like 60% of our users were actually the AV industry.
McGraw: Trying to figure out how to do tech stuff. [laughs]
Butler: They did troll that pretty hard.
McGraw: Yeah. I remember the HPGary hack too because all of my mail to Greg and Penny showed up on a Russian server. I went to search to see what I’d told them, and actually there was one mail I’m sort of proud of so I have to tell you about it. Greg had asked me for somebody’s cell phone number and I said “I can’t send that to you on this channel.” [both laugh] And when I read that on a Russian server, I was like “yes!”
Butler: That’s awesome.
McGraw: Good OpSec. But it’s a little weird to go see your email on somebody else’s server later.
Butler: I was mentioned a few times in those emails I think.
McGraw: Yep. I remember.
Butler: Greg and I were having some differences in opinion on some things.
McGraw: [laughs] I never use that channel for anything important.
Butler: It was funny because I was told not to read it by our CEO at the time when I was working at Mandiant. So I didn’t read it. And then, I walk into the restaurant a day or two after it happened—when it was leaked on the internet. The people there got up from the table—I was meeting a bunch of security people—and they got up from the table and they called me what Greg had called me. He didn’t call me to my face. He told like Penny and Bob and other people that I was this big “” and they all got up and yell that at me and I just look at them like “what the—“
McGraw: That is hilarious, man.
Butler: And they’re like “you haven’t read the emails?” And I’m like “no, I’m not allowed to.” Then they were like “oh, that’s what Greg said.” [both laugh] But we’ve hugged and made up since then.
McGraw: Greg’s a great guy. It’s hard to stay mad at him for too long.
Butler: Yeah, I don’t take anything personal. [both laugh]
McGraw: So, you started your career at the NSA before moving on to Mandiant, HPGary, FireEye, and all that stuff. Can you compare and contrast those worlds? Is it just like utterly different? I mean, HPGary versus Mandiant must be pretty different too, really.
Butler: I was brand new to computer security when I was hired at the NSA. They had a great program called SNIP at the time as an incentive, because you really weren’t making commercial dollars. To compensate for that, they have great training and they would—actually the year I joined up in ’97, they would pay for your master’s degree, or at least a full year to go get your master’s. So, that was a great program and that’s when I went to UMBC to get my Master’s in Computer Science. It was very interesting. The IAD (Information Assurance Directorate) was responsible for securing the government’s computers.
McGraw: Sure, that’s where the common criteria stuff lives.
Butler: Right. The hard part though, for me at the end of the day, was that they really didn’t have the authority over any of the government institutions that they were asked to help.
Butler: So they couldn’t evaluate NASA’s systems or even DoD systems unless they were invited. And, even if they were invited, they couldn’t mandate that the patches were in place or “don’t use this software because it’s got all these vulnerabilities” or anything like that.
McGraw: You know what, Jamie, it’s actually gotten even worse after the Snowden stuff. Now people are even less trusting of the NSA and they don’t want them to help the rest of the government, ironically. So, it was bad then and now it’s worse. I was just talking to some NSA guys last week and we were talking with a guy from Homeland Security who said “we sure could use some help, but last time we tried to get some help, we got in big trouble” and it was rejected from the body, pretty much. Interesting stuff.
Butler: The last three years I was there, I was in SigInt and that was a lot of fun. I worked with some brilliant people in my time there. I was always learning new things. I was still fairly young to the industry. I kind of felt more of a sense of purpose because you were trying to actually fight terrorism and different spies and things like that. That was intriguing. I really liked that part. But, at the end of the day, after five years, I kind of got a little bit burned out on it and decided I needed to go try something in the commercial sector and so I decided to leave. But, sometimes when college kids come up to me, or even high school kids, and are like “hey, I’m thinking about a career in cyber security. What do you think I should do? Where should I work? I got this offer from the NSA” and whatnot. I always encourage them. If they’re not a specialist yet, and don’t know exactly what they want to do, the NSA is a great training ground and they invest a lot in their people, or at least they did when I was there. So, I usually encourage them to try that road. It’s not a bad road to go.
McGraw: And then you worked for Mandiant for seven years, which is a long time, before it was bought by FireEye.
Butler: Yeah. Exactly. I had been CTO at a small start-up between HPGary and Mandiant.
McGraw: Komuko, is that right?
Butler: Yeah, they were going in a different way than what I’d originally signed up to do. I talked to the CEO and I decided to leave after about eight months or so. So, I was unemployed. I was getting married in four months, had a honeymoon to pay for, and a ring to pay for, and all that stuff. I didn’t have anything to go to. I was talking to a few companies in California but everyone wanted me to move to California. I’m kind of an East Coast guy, or I have been my whole life.
McGraw: Heck yeah, man. Go for it. Me too.
Butler: I was like “we have the internet and a few things like that today. You can do this remotely. You all have some offices here on the East Coast. I can go to one of those.” They were like “No. We really don’t want you to work remotely.”
McGraw: We try to keep our researchers in a special room.
Butler: One of the companies I was talking to at that time had had their source code stolen. So they were obviously really concerned about any remote connection to the office.
McGraw: I remember that story. That was hilarious. I mean, the silly thing was that you could reverse the dang thing anyway. Having the source code out was not that big of a deal.
Butler: I was introduced to Mandiant by an employee who used to work for my wife. He was like “hey, you should come interview at Mandiant.” I was like “I’ve never heard of you all.”
McGraw: You didn’t know Kevin? That’s interesting.
Butler: I didn’t know Kevin, even though he was Air Force and DoD, and that kind of thing in the past. I didn’t know him at the time. This was like 2006. He wasn’t extremely vocal back then.
McGraw: He was running around doing a lot of tutorials and stuff. He’d been doing that for several years.
Butler: He’d been at Foundstone and stuff like that. Anyway, the guy invited me to dinner with Kevin and the VP of Engineering, Dave Merkel. I was like “I’m unemployed. I’ll go to dinner with you. You’re buying right?” [both laugh] And they were like “yeah.”
McGraw: You got any good tickets to Hawaii for me? [both laugh]
Butler: So, I went to dinner with them and they gave me kind of a David and Goliath story. They were this small start-up and they were trying to take on the incumbent and things like that. That drew me to the company and what they were trying. They wanted a Windows end-point and I’m like “yeah, I kind of know a few things about that.” So, yeah. I started there.
McGraw: That was a heck of a ride, man! That thing went out for a billion dollars. Craziness.
Butler: Yeah, it was nuts. I’m very fortunate that it turned out the way it did.
McGraw: So when you joined it was probably still pretty little and grew really fast as heck.
Butler: I think we were about 26 or 27 people when I joined. I remember, maybe the first year I think our revenue was under even five million. It was like really small.
McGraw: And then, at the end, it was a big company. And then it got bought by an even bigger company and now we’re not sure what’s left.
Butler: We were about 500 people when we were acquired. We just like two years earlier had taken on 70 million of funding from a few VCs in the valley. So, we’d exploded from going from like three years earlier from 150 or so to 300 and finally to 500.
McGraw: That’s amazing. You managed a bunch of researchers there. I guess, what, 19 people? Something like that? What’s that like? I mean, do you like managing people? We sort of talked about that at the top, but researchers—I’m using that term in the hacker-boy sense and not in the science sense, I must say to my listeners—researchers are interesting people, to say the least. So, managing them is, I don’t know. How’d that go?
Butler: It went pretty well. There were a few odd things that happened here and there that I can’t go into, but… [both laugh]
McGraw: Aw, come on. It’s only 10,000 people that are going to hear this.
Butler: Well, I did have these two employees that worked at a remote office. One of them called me up and he said “hey, I almost got in a fight in the parking lot with this other employee.” And these two were kind of at each other’s throats a little bit. And so the older gentleman told the younger guy, who actually had a black belt in karate, that he wanted to go to the parking lot to fight. And I was like “oh my gosh.” They are telling me this over the phone individually on our one-on-ones. I’m like “you realize you can’t do this. Actually, I should do the responsible thing and take this to HR, but then it’s going to go in your record and be a big mess. Just chill out. Don’t threaten to fight anybody anymore. You two need to sit down with me and we need to air the grievances and talk face to face as gentlemen and get this behind us.”
McGraw: You’re such a Southern boy. [both laugh] Harrisonburg’s coming through, man.
Butler: Yeah. So, that was one of the biggest things that happened while I was there, but normally it goes well. I think that I consider myself a researcher/coder, just like the people that work for me. It’s a lot of comradery and I think you set the right vision and you get hard goals and you work as much as anyone working for you does, or maybe more. You set the right example and it goes pretty well.
McGraw: Yep. I think that’s very good advice.
I’ve got one last question for you. Do you think a track on security engineering would be useful at Black Hat? I know you’re on the committee over there.
Butler: Yeah, I think it would. We have one that not many—or not any—people submit to. I think it’s been called out once or twice in the call for papers. It’s kind of a software engineering track that covers security issues, you know, and coding and things like that.
McGraw: Yeah, that was the thing that Shostack tried to get booted.
Butler: Yeah. So we have that, but we don’t get any submissions to it really. I think maybe we had one that could kind of qualify for that, maybe at the Asia show that’s coming up. I don’t remember if it was selected or not. We’re kind of still in the process of that.
McGraw: I think if you made a big deal out of that, somehow, that that would be great. You know how I feel about building stuff. I think that’s our only solution here, otherwise we’re going to be stuck on an endless hamster wheel of pain.
Butler: And we have so many new people coming into the field every day. There are these incubators out there that take someone for three months and they’ve never written code before. At the end of the three months, they’re actually in a state that they’re good enough—you can hire them for your company. We’ve done that here at Endgame. I’m not sure about the previous places I’ve worked, but they don’t inherit that heritage from secure coding and all these other things that happened over the last decade or so.
McGraw: No doubt. Plus, things are evolving so quickly with dynamic languages and all the web crap, you know?
Butler: Oh yeah. You can’t keep up with it all.
McGraw: Well, you have to in secure engineering. That’s the challenge. So, last thing from total left field. I know that you’re into humanitarian causes. What’s your favorite one, or the one that you’re most interested in now?
Butler: I follow Hackers for Charity and support them. Johnny has moved his family and everything to Africa. He’s trying to help the local people and train them so they can be self-sufficient and do computer-type work and so forth. I just look at that as an incredible example. I can’t imagine giving up my house and my car and—we don’t have kids, but—his kids are going to school in Africa and so forth. That’s pretty amazing and pretty selfless of him. So, I really like that. Locally, I support a charity called IJM (International Justice Mission). They fight human slavery and sex trafficking all around the world. That’s a good thing I like.
McGraw: Cool. Well, it was good talking to you, Jamie. Thanks for your time.
Butler: Yeah. Thanks a lot for the time.
McGraw: This has been a Silver Bullet Security Podcast with Gary McGraw. Silver Bullet is co-sponsored by Cigital and IEEE Security and Privacy Magazine and syndicated by SearchSecurity. The September/October 2015 issue of IEEE S&P Magazine is devoted to the economics of cybersecurity. The issue features our interview with European cryptographer Bart Preneel.