Show 128: Lesley Carhart Discusses Incident Response and Digital Forensics

November 29, 2016

Lesley Carhart is the Security Incident Response Lead at a large corporation in the Chicagoland area where she and her team work with digital theft, misconfiguration, and hacking issues. She has 17 years of experience in the IT industry, eight of which focus on incident response and digital forensics. Lesley holds a BS in Network Technologies from DePaul University. She is an active writer, speaker, and works as a member of CircleCityCon staff.

Listen as Gary and Lesley discuss the evolution of computer security, incident response, digital forensics, security engineering, security certifications, and more.

Listen to Podcast


Gary McGraw: This is a Silver Bullet Security Podcast with Gary McGraw. I’m your host, Gary McGraw, CTO of Cigital and author of Software Security. This podcast series is co-sponsored by Cigital and IEEE Security and Privacy Magazine. This is the 128th in a series of monthly interviews with security gurus, and I’m super pleased to have today with me Lesley Carhart. Thanks for joining us, Lesley.

Lesley Carhart: Thanks for having me.

McGraw: Lesley Carhart has spent the last 17 years in the IT industry, including eight years focused on incident response and digital forensics. Lesley is an active speaker and writer in the Chicagoland InfoSec community. She’s also involved in running CircleCityCon in Indianapolis. Lesley has a BS in Network Technologies from DePaul University in Chicago. In her free time, Lesley studies three martial arts and is a competitive pistol sharp shooter. Lesley is known as @hacks4pancakes on Twitter (with the number four). Although, you probably are already following her gigantic feed. So, welcome to the show.

Carhart: I’m glad to be here.

McGraw: Before we get started properly, I just have to ask you this as a Virginia country boy: any relationship with the “Carhartt” Carhartts?

Carhart: You know, if I had one more “t” at the end of my name I would be very wealthy right now.

McGraw: Are they double “t” Carhartts?

Carhart: I wouldn’t have to do this life of infamy and security.

McGraw: Okay. Fair enough. Well, I’m from the poor branch of the McGraws, so I totally understand.

What got you started in information security? And, how have you seen InfoSec evolve in the last decade since you’ve been really focused on that?

Carhart: I actually got started in the security space pretty early on. I started programming when I was around seven or eight, on a 286. I got my first job in IT when I was 14 years old as a SQL developer because it was the .com boom and anybody who was decent coding could probably get work as long as they could interview well. So I had an early start in the field, and that’s how I really got involved in the Chicagoland hacking community as a teenager. And things don’t always turn out as you plan. As you know, the .com bubble burst and we all had to go to our “plan Bs.” For me, it was enlisting after high school, and I did technical work in the military too. But I kept that really deep-set interest in taking things apart, and learning how things work, and how to make them do different things that I think is really integral to being a hacker. I did have an interest from very early on in computer forensics, even when nobody knew what it was. I would go to colleges and I’d be like, “I want to study computer forensics.” Computer science people would send me to the law enforcement curriculum right there.

McGraw: Exactly.

Carhart: And the criminal justice professors would send me back computer science. So I’m lucky to do the job I wanted to do.

McGraw: Wasn’t Dan Farmer’s book one of the first in the forensics field, or am I just crazy?

Carhart: It was. I bought every book that you could buy on Amazon when Amazon only just sold books—all four of them. Back then we were talking about very different things because it was all disk forensics. Even looking at memory was a theoretical thing. So obviously the field has changed drastically. It certainly changed from something you did on top of being a systems administrator. Or, if you were a security specialist in the rare niches, you did everything. You did everything from computer forensics to firewall management to responding to attacks. But now we’ve obviously become much more granular and we’ve become specialists in very specific fields, as it goes with any emerging field.

McGraw: Yeah. I want to dig into that later on in the show. But first, what in your experience, is the difference between doing network operations and technical work for the U.S. Air Force, versus doing that for a big corporation?

Carhart: I think that the challenges are very, very similar. Any large organization, you really have to have a good leadership team and a good idea of how to manage without micromanaging. And although government and private sector jobs may face different challenges, they still have the same basic needs and they’re still using similar technologies in dealing with some similar issues. I don’t think there’s really a big problem in moving from one to the other.

McGraw: You think it’s a natural movement?

Carhart: I think so.

McGraw: That’s interesting. And for those listeners who are not in security operations, can you let us know what the number one rule for incident response is for non-IR people?

Carhart: I would say that the number one rule for incidence response is “don’t panic.”

McGraw: Yup. That’s a great rule.

Carhart: That’s my personal favorite. I once did a talk on my top 10, but my number one pick out of all of those, I would have to say, is don’t panic. I mean, it’s a high pressure job. It’s dealing with triage. It’s dealing with high pressure, high risk environments, and it’s often doing it outside of normal business hours or for extended hours. And other industries that do that, like emergency response and emergency medicine, they have tools to deal with that and we don’t. So definitely staying calm and not panicking in that type of high pressure environment is critical.

McGraw: Okay. And then the same thing for forensics. Is there a number one thing that those of us who don’t do digital forensics need to know?

Carhart: Sure.

McGraw: Probably don’t step on the evidence.

Carhart: Yeah. That’s a fundamentally critical routine in custody and evidence handling. But I’d also say the basic rule of investigation is don’t try to make your evidence fit your conclusion. You build a conclusion based on facts and evidence, not the other way around. And I see a lot of people doing that wrong. They are certain that something had happened and they try to make all of the evidence in their computer forensics investigation fit that conclusion, which is wildly inappropriate and totally against the laws of investigation.

McGraw: Yeah. That’s an incredibly important point. Things like attribution require the same sort of evidence instead of a “jumping to” of conclusions. And I worry that there are some people who think that their foregone conclusion can be justified later, but in instantaneous cyber war that’s a really rash and not very logical approach.

Carhart: It’s so frustrating when computer security people keep saying, “Attribution is hard, attribution is hard.” But it is very hard. It’s possible to a degree, just like any investigation. You’re trying to draw the most plausible conclusion beyond a reasonable doubt. And that’s what you’re doing in attribution. You’re taking all of the evidence appropriate to make a conclusion that is solid beyond a reasonable doubt, to a reasonable person. And what that threshold is is going to be dependent on what the reaction is—especially when you’re talking about warfare.

McGraw: Yeah. And time too. I mean, you can’t just figure that out in 32 seconds. That’s not how it works.

Carhart: Absolutely.

McGraw: Yeah. So I think the people in Washington still need some more of that Chicago thinking, if you could help with that it would be great.

Carhart: I certainly would endeavor to do so. Nobody has asked me.

McGraw: Right. Let’s talk about education for a bit because I know that you spend time doing outreach in classrooms.

Carhart: I do.

McGraw: And I’m wondering how early should computer security education begin in your view.

Carhart: I think that it should be taught very, very early on. And I’m certainly biased. As I previously said, I have a very, very traditional background for a secure hacker which is not at all normal. So I hate talking about my background because lots and lots of very proficient hackers started in their 20s. Or, didn’t own a computer until they were a teenager or an adult. I think that it does give you a fundamental advantage though, because a lot of the college programs that are out there in computer security, and the training programs, make assumptions about the skills that you are coming with.

McGraw: Right. So you need some base to work from, really?

Carhart: Yeah. And if you don’t have that you are already at a disadvantage. So the sooner we can expose people, and offer them the ability to get interested in computers and in computer security, the better. And it’s also a matter of dealing with, you know—we still have the concerns with kids getting in trouble on the Internet and getting involved in illegal hacking, and that’s never going to be solved by saying, “Let’s send them to jail.” It’s a matter of being involved in their education with computers and them learning about computer security. If you say, “Never learn how to take apart a computer because you might end up being a law breaking hacker, a blackhat,” teenagers are obstinate and they’re just going to go off and do it in their own time, without their parents’ knowledge.

McGraw: Yeah. So maybe what you do is you teach how to take stuff apart and how to break systems, but you also teach some ethics.

Carhart: Yeah. And you have an opportunity to do that. Just saying, “No. Don’t learn how to do this,” is costing the opportunity to teach people how to do things responsibly and ethically.

McGraw: I totally agree with that. So there’s this thing that I call the “NASCAR effect” that happens when all of the media focuses the attention on spectacular hacks, and crashes, and data breaches. And it’s all about explosions and dumpster fires. I’m wondering how the NASCAR effect helps create this infosec echo chamber, and some of the problems that we were just talking about.

Carhart: Well, it happens across all industries. There can certainly be harmful, hurtful effects from computer security. And we have the temptation to play into that NASCAR effect. We want people to pay attention to that. And it’s not just that, it’s that we have legitimate concerns about security, and the potential for more kinetic effects with the Internet of Things and more industrial control systems being networked. So there are real risks out there. When we try to get the attention of the media, sometimes I think that computer security people who deal with the media feel like they have to play into that NASCAR effect. They have to say, “You could take a hospital down with this.” When the real risk is like in the future. There could be potential attacks on infrastructure, and a potential thing that we should be fixing now proactively. But that doesn’t work. Talking about theoretical things in the future doesn’t work to get the attention of the media in their opinion. So they start talking about, “Oh my God, somebody hacked a Jeep going 70 miles an hour.” (I’m not saying that’s right or wrong.)

McGraw: Charlie likes it.

Carhart: Yeah. For Charlie it worked. It got attention on the case. I know that there’s a wide range of opinions on that, playing into that NASCAR effect, but there’re problems on both sides. I think that the reason security people are playing into it is that it can’t find a way to get media attention without doing it. I’m sure it’s happening in other fields too.

McGraw: I’ve been getting media attention for security for 20 years. And it is very difficult to turn a story into, “Hey, let’s build this stuff properly. Hey, let’s do some engineering.” As opposed to the big bad boogie man and explosions. So I completely understand where you’re coming from. It would be nice if we could figure out a way collectively to do a better job at that, but I’m not sure with new people coming into the field all the time, trying to get attention for yourself is what everybody seems to do.

In your view as kind of a leader in the security space on Twitter, does Twitter help or hurt with that thing?

Carhart: I think that it helps us build a community and act as an organism instead of it’s a bunch of individuals. I think that social media has been great for that.

McGraw: That’s good.

Carhart: I think that a lot of security people are using it in very good ways to do community outreach and education. There’re so many security people who you would think are rock stars—are untouchable in terms of other fields—who are really willing to sit and talk and mentor people, and educate people. We have a great community for that. If social media helps us become more a cohesive community, that’s fantastic. And so far, that’s been the case in my opinion. I mean, there might be a time in the future where it drives us more apart, but at that moment it’s still a great outreach tool and educational tool.

McGraw: We’ll be right back after this message.


If you like what you’re hearing on Silver Bullet, make sure to check out my other projects on There you can find writings, videos, and even original music.


McGraw: There’s this tricky bit I wanted to get into a little, which is this trend towards professionalism. Generally speaking, a good thing can also negatively impact diversity in a field. And so when I got started in computer security in 1995, everybody in the field came from all sorts of different backgrounds. We had biophysicists and we had cognitive scientists and we had people without college degrees, and we were all on the same field kind of inventing it in the mid 90s. And since then things have professionalized a lot. And so the backgrounds are getting to be somewhat more similar and that is tricky for me to parse. I don’t understand if that’s good or bad, but I’m really interested in your thoughts about that.

Carhart: It’s a tough question. I’m conflicted about it too. I certainly haven’t seen many degree programs yet that are teaching how to really be a good creative hacker at heart. There’re a lot of programs that are supplementing those skills, while for people who already have that drive and already have the interest, that’s really necessary to be a good computer forensic investigator or a good white hat penetration tester. So there’re programs that can teach you skills, but I haven’t really seen many that really teach you to have that drive to innovate.

McGraw: Yeah. And if people are not coming with their crazily diverse backgrounds like before—because now you’re required to have a BA in security or whatever. I don’t know. We might lose something as a field. Even while we gain something. So I don’t think it’s all cut and dry one way or the other.

Carhart: I’m always a strong advocate for getting some kind of formal education, if it helps you grow as a person. I won’t just count the benefits to there being degree programs that hackers can get into and find benefit in. In terms of things like business communication skills and writing papers—

McGraw: Speaking English.

Carhart: Building interpersonal skills, yeah. And that’s what’s really important. And those are things that college can really help with. I still haven’t found a college program that can make you a hacker, a good hacker.

McGraw: Yeah. I don’t think I have either. For a while UC Davis was pretty good at that one. When Matt Bishop was really active in the space, but that was 10 years ago. Now he’s doing other stuff as a senior faculty guy.

Let’s talk about certifications because these are sort of related. You have lots of them. You have GCIH, GREM, GCFA, a GPN, and I don’t know what any of those are.

Carhart: Yeah. I do have a lot of those after my name, but it does not go on my business card.

[both laugh]

McGraw: There are probably even more. So why? I mean, what was helpful for you with that stuff? Were you trying to demonstrate experience that you already had, or did you learn something? Or you know, I’m interested to know your view on that, the whole certification issue.

Carhart: I think that as long as you find certification classes that teach you something that you didn’t know that you are interested in learning, it’s good to find an employer that’s willing to support you in getting those.

McGraw: Yeah. Very nice. Yeah.

Carhart: I don’t think that they should necessarily be requirements for jobs, unless they are for some compliance standard. But when you’re evaluating, and you should be evaluating the company that you’re applying for—when you’re asking them questions, one of the questions you should be asking them is about the training opportunities, and the certification opportunities that they will guarantee to offer you in that role. Any good security job should offer you the potential to learn and grow on your own terms to some degree.

McGraw: Sure.

Carhart: Whether that’s going to conferences, or doing public speaking, or going to certification programs on topics that interest you or you think are well regarded and interesting. So, I think that I wouldn’t tell everybody to go out to college right now, and get a bunch of very expensive certifications on their own without a sponsor right now. But when you’re looking at those jobs, there are good certification programs out there that will teach you technical things, and they are not necessarily security certifications either. If you’re studying PowerShell, maybe you want to go take a PowerShell class, or maybe you want to take a class on an obscure database format that you’re running into, and you want to do more penetration testing on. That opportunity should be there for you in any work environment.

McGraw: In your own experience, have you learned more pursuing certifications or going to conferences yourself? You’re good at both.

Carhart: I don’t want to promote a specific vendor on this interview, but I did take time to carefully research the certification program that I did go through, and make sure that they are well regarded through the community and that they had a curriculum that included the technical stuff that I really wanted to learn. I never got a certification just for the sake of getting a certification, except for a couple of things that were required for compliance.

McGraw: Wait a sec. So I’m going to push back on that. So I think it’s totally fine say who you thought was good or who did a good job for you if you feel like it. So who was it?

Carhart: If you think that’s appropriate. I have been a proponent of SANS and GIAC on my blog before.

McGraw: Cool.

Carhart: I don’t think it’s a secret that I think they have good programs. They’re very expensive and they know that, which can be a big barrier to people. But their programs are very worthwhile, especially their programs with their core curriculum teachers who wrote the book.

McGraw: Yeah. I’m totally with you. I think there’s some extremely good people there. And when people ask me, they say, “I’m a technical person but I’m just getting started in security, and I don’t really know where to start.” Ill recommend some books and tell them to go to some SANS stuff. So I think that’s totally fine. And of course your mileage may vary. It has to be suited for you. I like what you said about the willingness of an employer who pays for these things. That shows that the culture that you’re joining or that you joined cares about you. I mean, cares about developing your capabilities. So that’s something to look forward to when you’re interviewing out there, I guess.

Carhart: And that’s important even if you want to stay at the same employer for a long time. You still want to grow as a technical professional. You never want to stagnate. You want to keep up to date on current events, and you want to keep up to date on current technologies, however you chose to do that.

McGraw: Yup. So now I have a question that, you know, there’s a big debate over whether I should even ask this question, and I’m going to ask it in a funny way and then you can tell me whatever answer you feel like. Any advice for men in security to help women in security feel welcome in the field?

Carhart: Wow. For men in security. Human beings are human beings. Don’t treat people like there’s another, no matter what gender they are or what cultural background they’re from. People have the same basic needs and wants. And if they want to be a hacker, we all have a fundamental prize. It’s the feeling in your gut or in your heart to learn how things work, and how to do it, and make them do different things. That’s being a hacker. If you see somebody at a computer security conference who’s trying to learn, they have that drive too and that’s more important than any physical distinction between people. So, I wouldn’t necessarily ask men to treat women differently, I would ask them to try to treat hackers like hackers.

McGraw: Very nice. I think that’s eloquently put. Thanks.

Carhart: Sure.

McGraw: Then the last question in normal question land. Do you worry about the problem of testing security in, versus building security in? Let me try and explain that a little better. That is, the problems around security engineering; do those worry you as a security operations professional or is that somebody else’s job, or what’s your view on all that stuff?

Carhart: Yeah. That’s a huge problem, especially if we get into, as I said previously, more IoT and ICF data. Building security in is crucial as opposed to tacking security on, and they’re trying to secure something that is fundamentally insecure, or monitor something that is fundamentally insecure, it’s incredibly complicated and almost impossible. You do your best and you make risk decisions. But without those basic concepts of security being involved in the engineering product or service, it’s very, very, very difficult.

McGraw: So I knew you were going to say that. And I think there is a cadre of enlightened security operations people who think that way. But, I also think that we have this propensity to slide into, “Well, let’s just pen test it and we’ll fix the stuff the pen testers find, and we’ll test our way into security.” That is crazy. So I don’t know. Can you help to get the other operations people to figure that out?

Carhart: If you find one little bit of an iceberg sticking out of the ocean, that means that there is a very large iceberg underneath it waiting to sink the Titanic. So, if your pen tester gets out there and looks at your product that’s built with no security or secure practices in mind, and they find a hundred different vulnerabilities, and they’re able to exploit some of them, that means that in the next five years there’s going to be a thousand more found in that product that we don’t know about right now. Next week’s zero day, that product is going to continue to be vulnerable even though you’ve dealt with the superficial problems that are visible right now.

McGraw: Yeah. So really the way out, especially I think from an economic perspective, is to try to design things so that they can be operated in a secure fashion later. But that sort of runs counter to the NASCAR effect that we were talking about before too. Because then you don’t have the spectacular explosions that the press covers, and everybody gets all happy and then the field, I don’t know, becomes like an engineering discipline. When is the last time you heard about a bridge being built in the newspaper?

Carhart: I think that you just discussed the fundamental problem with human society, not necessarily just security.

McGraw: Yeah. Well, maybe we need some journals or security publications about good bridges being built. I don’t know. I don’t have a good way—how do you appeal to younger people that are interested in security, and teach them that it’s about discipline in engineering, as opposed to just blowing stuff up?

Carhart: I think that there’re a lot of great creative programs out there right now. I just saw somebody build a Minecraft security model where the players have a few minutes to build up a firewall and catch vulnerabilities and then the zombie hackers come out and they have to defend against it.

McGraw: I love it. That’s cool.

Carhart: Teaching young people how to—and you’re going to have to go look up the speaker who gave that talk so you can give them credit, but—

McGraw: I’ll totally do that. We’ll put a URL in the

Carhart: Yeah. But it was fantastic. I think that there’s a lot of different programs out there trying to think of creative, fun ways for kids to think about what a vulnerability is and how to build things that are smartly engineered. And we just have to be creative and try to get kids involved in that because they’re smart. They’re little people, they have good ideas of their own.

McGraw: So last question which has nothing to do with anything. What is your favorite pistol to shoot?

Carhart: Oh, my favorite pistol to shoot. I compete with a Ruger Mark III—

McGraw: Ruger makes great stuff. And so you shoot competitively?

Carhart: Yes.

McGraw: Great.

Carhart: I’ve actually run a gun club and I’m involved in training people how to pistol shoot.

McGraw: Very cool. I like shooting pistols too.

So thanks for your time. It’s been an incredibly interesting conversation.

Carhart: Yeah. Thank you very much. I was really happy to be here.

show 128 - Lesley Carhart