Gary McGraw: This is a Silver Bullet Security Podcast with Gary McGraw. I’m your host, Gary McGraw, CTO of Cigital and author of Software Security. This podcast series is co-sponsored by Cigital and IEEE Security and Privacy Magazine. This is the 123rd in a series of interviews with security gurus and I’m pleased to have with me today Yanek Korff. Hi, Yanek.
Yanek Korff: Hey, Gary.
McGraw: I have known Yanek for a long time. Yanek Korff is currently a Consultant for Korff Consulting, LLC. He’s also the owner. Yanek was previously the Vice President of the Mandiant Computer Incident Response Team. He’s also worked at AOL as the leader of the Security Engineering Team, at Bell Atlantic, and also at Cigital a million years ago. Yanek has a decade or more of experience in security operations, development, and infrastructure. He also established AOL’s PKI. We serve together on the Advisory Board of Ntrepid. Yanek has a BS in Computer Science from the College of William and Mary and a MS in MIS from the University of Virginia. He lives in Virginia with his family but he’s in California today.
Korff: How are you doing?
McGraw: I’m good. I hope your trip out there wasn’t too bad.
Korff: It was pretty uneventful. It was a bit of a last-minute trip, so I caught the much-wanted middle seat but I survived.
McGraw: Oh, you lucky guy. And probably on United too, huh?
Korff: Absolutely. My favorite airline.
McGraw: I think they’re beginning to realize that they suck, which is the first step towards not doing that.
Korff: Yeah, it’s interesting. I actually ended up needing to call support due to some IT issues but that turned out to be a pleasant experience and I was shocked.
McGraw: Cool. Well, you started out, speaking of pleasant experiences, as a systems administrator. Then you did security ops and ended up as an executive in charge of services around network ops and incident response. So, in this career (you’ve been doing this for about 20 years), what’s the most important thing that you learned about security or about working in the field?
Korff: The most important thing I learned is probably, gosh, can I say this without sounding corny? It’s about the people.
McGraw: That’s not so corny.
Korff: Technology works for a limited amount of time and if the people can’t adapt and throw out the old technology and bring in new stuff and change, it all falls apart.
McGraw: But I thought systems administrator people just didn’t like those darn pesky users.
Korff: Oh, I still don’t.
McGraw: But they’re people, Yanek. Come on.
Korff: They are, they are. I have a love-hate relationship with people. It’s hard to explain.
McGraw: So it’s about the people. How about coming up through having to use other people’s operating systems and stuff and gear while monitoring for security and handling incidents? Was there anything that struck you as something that was counterintuitive that you learned?
Korff: I think one of the big turning points for me actually, funnily enough, happened at Cigital. We were putting in a new firewall and really struggling with it. Things were not flowing past it. I was green. I didn’t know how to troubleshoot any of this stuff. I was working with a more senior guy and eventually we sort of ran out of ideas. I called my boss. We talked him through everything we’d done, all of our ideas. And he said one thing I will never forget that really helped us out. He said, “Well, good luck.” That was it. That’s all the guidance we got.
McGraw: So you learned that you can use that magic catchphrase as an executive? Is that what you’re saying?
Korff: No, no. The thing I learned there was, when you’re stuck and you can’t pass the buck but you’ve got to make something work, you figure it out. And if you keep answering other people’s questions with just, “Here’s the answer,” and they don’t need to work for it, they grow a lot less. That was super, super valuable. I don’t know if that was the intended lesson but that’s what I took away.
McGraw: Was that Scott Marks?
Korff: That was, yeah.
McGraw: It probably wasn’t the intended answer. It was probably beer 30.
Korff: It was probably 11:30, I think, at night when we were at that point.
McGraw: That’s hilarious. Let’s talk about SOCs and SIRTs for a minute. So how much of a SOC and SIRT is people versus automation these days?
Korff: I think that’s very highly variable and it really depends on what you’re trying to do and the kind of SOC you are trying to build. I built a managed security service, so that’s naturally going to be multi-tenant and it depends on what you’re trying to slice. We always looked at things pretty much from a threat-centric perspective. If you’ve got opportunistic threats that are on your network and consuming your IT team’s time, that’s one problem. If you’ve got targeted threats who are very interested in your company’s intellectual property or want to commit fraud, that’s a different scenario. The latter is a lot more people-intensive than the former. But I think if you look at the managed security space today, you’ll find they’re really applying people to technology problems in a lot of cases. I’ve been kind of disappointed with the way that space has been evolving.
McGraw: In a sense that the automation isn’t what it could be? We still need the people but automation needs to be improved.
Korff: Yeah, absolutely.
McGraw: What do you think, are companies better off running their own SOC, or outsourcing, or some combo package, or it depends, or what’s the answer to that?
Korff: If you look at the, we’ll call them the security one-percenters, the guys who have massive security budgets and have somehow an access to security talent that’s relatively unlimited, sure, you can buy the right pieces of technology, you can put them together and grow your own security operations center. And most importantly, if you can provide interesting bad guy stuff on your network all the time, then you can even retain those people. The security people really need a constant stream of entertainment, which most companies refer to as “bad news.”
McGraw: So outsourcers get better people because it’s less boring?
Korff: It’s tremendously less boring. I talked to new employees back at Mandiant 90 days after they had started and consistently would hear, “Man, I have learned more here in the last three months than I did in the last three years of my last job.”
McGraw: Yeah, we hear that too. Of course, we’re not really an operation, so there’s always stuff to do in security engineering because you’re building new things. Software security is different that way. But operations is just as important and it does get boring. If nothing happens, you’ve done your job. Ironically, if nothing happens and you go and ask for an increase in budget, then they tell you that nothing happened and they cut it.
Korff: That’s the funny thing about systems administration, where I came from, right? If you’re doing a great job, nobody notices you. But if you’re not doing a great job, everybody notices you for the wrong reason.
McGraw: Hard to be right.
Korff: Yeah, it’s a no-win.
McGraw: What do you think we’ve learned over the years from incident response in terms of computer security?
Korff: I hope that we have learned, and this may not be exactly what you’re asking, but I hope we have learned that one response does not fit all. For a long time, we worked with customers who believed you detect and you re-image the machine, and that is incident response. Like, that is the process.
McGraw: Just clean up the mud puddle and keep on going.
Korff: Exactly. And it took tremendous amounts of time and education to get people around to the idea that, “You know, maybe instead we can ask some questions about how that got there, figure out if it’s anywhere else, and then come up with a more comprehensive plan depending on what we’re dealing with.” In the case of those targeted attacks, sure, if it’s an opportunistic one, clean it up. Whatever. You’re going to get another one in a month or so. Take a look at the last month’s worth of data and go find some root causes for that and address it. So I think I see more companies out there that are taking that kind of an approach or at least are open-minded in that direction. I’m cautiously optimistic that we’re headed in a good place.
McGraw: I think you’re right about that and I think that actually informs things like software security, which are in some sense even more proactive. If you wonder why would you bother to build a system that’s more secure anyway? Well, if you’ve been through the incident wars, you know why.
Korff: Yeah. That’s definitely true, your initial vector for the guys—the true bad guys—tend to be the people with whom I may have mentioned I have a love-hate relationship. But once you get there…
McGraw: You can never really get a systems administrator turned around on that, can you?
Korff: No, you really can’t. It runs deep. But yeah, once you get past that first person as your vector, from there it’s system to system, and taking advantage of software vulnerabilities is a great way to move around.
McGraw: Yeah. So, now that you’ve developed pointy hair and some management chops, let’s talk about people a little bit more. When you’re hiring for your team, what’s the most important characteristic you look for and how do you look for it?
Korff: The corny answer, I guess, here is it depends on what team and what characteristics I’m looking for the people on that team.
McGraw: Pick one and make it concrete.
Korff: Let’s stick with the topic. So let’s talk about a security analyst and let’s say that the security analysts I want are going to deal with targeted attacks, not opportunistic ones. If that’s true, then I need curiosity first and foremost because when you see something that’s weird, you need to look at that and say, “Huh, that’s interesting. I wonder…” And then they’re going to want to take some action. Beyond that sort of basic curiosity is like a level of persistence where you just really don’t like unsolved mysteries. It’s sort of intolerable, and you’re creative, which means you adapt to different situations. So those three are probably the top ones for that kind of a role.
We’ve made mistakes at hiring folks who are more process-oriented. They like to know what the next step is and that didn’t work. That very directly did not work in that kind of environment.
McGraw: Right. So more of an engineer and less of a creative type, say?
McGraw: So how do you find people like that? In some sense, interviewing is like going on one date and deciding whether to get married, which is always a weird thing.
Korff: It’s certainly hard and I have yet to hear of anybody with a 100% success rate. But we did have a pattern we followed when it came to hiring and it really stemmed down to asking people to tell you about themselves. Like, “Tell me your story, where you came from, what you did, and all the changes you went through. Why did you do that? What prompted you to go from one job to another? What was not interesting about the last one? More interesting about the next one?” Through that story, you end up getting most of your questions answered, I found. You’ll find the folks who are like, “Yeah, I learned a whole bunch of stuff there and I felt like I had sort of mastered that. I got this other opportunity. I was talking to a buddy of mine and they had this problem they hadn’t figured out how to solve it yet. I thought that was really interesting, so I went over there,” and you’ll feel that enthusiasm come through.
McGraw: Right. So you’re looking for passion. You’re looking for enthusiasm and it’s okay if somebody moves on to something more interesting. As long as you’re a manager, you can make sure they always have something interesting in front of their nose if you’re doing it right.
Korff: Yeah, and frankly I’m not—and this is a little bit unpopular, but I’m not hell-bent on keeping people forever. I’ve changed jobs. I don’t want people to be stuck in a job that they don’t like. I had an employee years ago who came to me and said, “I don’t like my job.” That was the beginning of our one-on-one.
McGraw: Say, “Okay, you get a new one.”
Korff: I mean, kudos to her. That’s a ballsy thing to do. And we had a really good conversation, sort of figured out what the issues were, and they were issues that I was not going to be able to fix for at least two years between the shape of the business. So I was fully supportive when she changed jobs. It was very sad to see her go but happy that she was moving into something she enjoyed.
McGraw: Yeah, and in some sense, if you going to change the world like I’ve been trying to do in software security, you’ve got to expect people to go out and to grow past your organization sometimes. And as long as they leave and they’re happy with their experiences in the past, they can help spread the good ideas that they might have accidentally caught while they were working at your place.
We’ll be right back after this message.
If you like what you’re hearing on Silver Bullet, make sure to check out my other projects on garymcgraw.com. There, you can find writings, videos, and even original music.
McGraw: So, from a technical team member perspective what’s more important, technical skill or motivation?
Korff: There’s a strong temptation for me to say the motivation because that’s what is going to drive you, but you’ve got to be able to hit the ground running to earn the credibility of your peers. There’s a baseline from a technical skill perspective that you just can’t get around.
McGraw: Very well put. So you’ve got to have both. You can’t just be utterly motivated and clueless.
Korff: No. You’ve got to have some skills to bear and you got to understand where you are. You need to be very self-aware about where you are on that chain of expertise.
McGraw: So in terms of management skills, there’s this one skill that I like a lot that works incredibly well against geeks, and that is silence. Do you use that?
Korff: Yeah, probably inadvertently. As someone who both likes and dislikes people, I don’t necessary enjoy keeping up my end of the conversation. If I have nothing to say, I won’t say anything.
McGraw: You’re doing pretty well so far.
Korff: [“Attempts to move things along.”]
McGraw: Silence is often an uncomfortable social situation, so people will talk right into the gap, I guess.
McGraw: And that’s a good way of learning what somebody is really thinking or getting past the initial spin or whatever.
Korff: Especially if you don’t know where the conversation is headed.
McGraw: Yeah. So in some of your writing you talk about core values. And in some sense when you’re looking for employees, you want a certain set of core values: motivation, self-driven, curiosity, likes to solve puzzles, get stuff done. Those are all core values. Do you think that universities that produce geeks are instilling core values like that, or can you teach those, or where do those come from?
Korff: Man, that’s a good question. I don’t think it has anything to do with what you’re taught. I think it has a lot more to do with who you’re around and who you interact with. Because I perceive everybody models somebody else when you find those people to look up to and you take the best of the people around you. That’s where those come about.
McGraw: So there might be some people to emulate in the university setting but that’s just an accident of the people, not really the setting?
Korff: Yeah, I think that’s accurate.
McGraw: Okay. So really you should—I guess especially in graduate school look for the person you want to study with, not the university. The person matters more. I’ve always thought that for grad school. I’m not so sure you can do that in undergrad, though.
Korff: That’s a really good question. I think you’re unlikely to really be able to assess, before you go to any kind of institution, the kind of people you’re going to be with. That’s tough. I mean, there’s campus visits and sometimes you can visit classrooms and stuff but you don’t really get to know people until you spend a lot of time with them, right? You asked earlier about the interview process and you get the short date before you’ve got this long marriage. It’s similar. So, in many cases, I’m going to say that’s luck of the draw.
McGraw: I’m just going through that with my youngest son, who picked which college to attend through some mysterious process that neither his mother nor I understand. Hopefully it will work out. It worked out for the first one but it also seemed less mysterious for that one than the second one.
Korff: I’ve got a little time yet.
McGraw: It’s on my mind a little bit.
I’ve noticed in some of my really recent work, like stuff I did Tuesday, that technologists who go from alpha geek into management suffer from this thing that we’ll call the Superman Syndrome, where they want to tackle the hardest problem themselves. Let me think of an example. For example, a CISO who does his or her own threat intelligence-gathering like every morning because they don’t want anybody else to do that. As manager and as a geek yourself, as an alpha geek, how do you avoid the Superman Syndrome? What kind of advice can you give to our listeners about that?
Korff: You know, I got really lucky. I had that problem when I first took on management. It was a slightly different flavor but I went from being on a team to managing that same team. So I knew what we did. I knew the work. I could do the work and I was relatively well-respected in the team. As soon as I became the manager, I became the go-to guy for everybody on the team whenever there was a problem. “How do we fix it? How do I solve this? What’s the right way to build that?” And thinking I was helpful, I would answer those questions.
McGraw: This goes with your earlier answer. Very nice.
Korff: I know. Completely forgetting the lesson from Scott Marks, I struggled and everything actually went well. It was deceptive. I thought, “Well, this is going great. We’re getting a bunch of stuff done.” Everybody was happy. And then I went on vacation.
McGraw: Well, there’s your mistake.
Korff: Well, that’s a fair point.
McGraw: What happened? Do tell more.
Korff: Nothing happened. I came back and there was a line of people outside my door. All the projects were stalled. Stuff was broken. I had created this dependence that really didn’t need to be there. Those folks were smart folks. They could have done all of what was required without me there, but that sort of created this bad habit.
McGraw: So this Superman Syndrome is something you’ve seen in your own work?
Korff: Oh, yeah.
McGraw: We saw this at—I’m working with a bunch of CISOs and I see some super technical CISOs where that becomes an issue. It’s hard not to want to take over if you’re sort of a control freak alpha geek. What do you do besides just self-realization? Are there any techniques for not doing that?
Korff: I said earlier I was lucky. The luckiest part for me was we had a sister team at the time, so I was running a team of systems administrators. We had a sister team of developers and the boss of the development team, with whom I’d started a pretty difficult relationship with but it became a really good relationship, he left the company. And at the time he left, he said, “My team developers, they’ve got a lot of experience. I don’t know anybody else in this organization who can run this team. You’re going to have to hire for it or you can just give it to Yanek because he knows all those guys and I think that would work well.” And all over sudden, I went to doubling my team from 8 to 16 people, which is wholly unreasonable.
McGraw: Yeah. It’s a pretty big difference, right?
Korff: Yeah, it’s very different. And so I had to immediately shift my priories from one team’s mission to two teams’ missions and begin to start to think about structures that would help me run both of those teams. The answers were not obvious and it consumed all of my time because, at the time, I still didn’t have a ton of management experience. And I had no choice. I could not be the answer-giver.
McGraw: Talk about getting thrown in the pool. Here, swim.
Korff: Exactly, and what happened is the team survived. They realized, “Oh, gosh, he’s really busy with this other thing right now. I can’t do that. Let me see if I can figure this out.” And that’s really all they needed.
McGraw: Right, was to be weaned. They, in some sense, weaned themselves through necessity.
McGraw: Is there a similar switch between, say, managing 16 people or 2 teams up to 50? I think you had 50 when you were finishing up over at Mandiant.
Korff: Yeah, there were probably—I think by the time I left FireEye, I was close to a hundred but that was a little bit after the acquisition. Is there a difference? I think you take a lot of the lessons with you as you go. I know a lot of folks who have jumped management rungs, if you will, and I’m glad I didn’t. It’s a lot harder to do it the slow way, I think, but you learn a lot better lessons along the way. You have the time to make the mistakes that will sit with you for a long time.
McGraw: And you may have a mentor above you who can teach you some of those things too, right?
Korff: Rare, but yeah, that’s possible.
McGraw: If you do it the long way, presumably you’re learning the whole time?
McGraw: So if you had to characterize the difference in running 100 people versus, say, running that first team of 8 that you did 10 years ago, what would that be?
Korff: I don’t know that I have a crisp answer. It really depends on the shape of the team and the people. At the end of the day, you’re still running a team of people. It’s just a question of whether they’re managing people themselves or not. But your job is still to develop a high-trust one-on-one relationship, understand what their challenges are, and coach them through it without solving their problems for them and helping them grow. And whether you’re doing that to somebody who has direct reports or not direct reports, you’re doing the same thing. You’re just dealing with different problems.
McGraw: Right. Did they ever send you off to management school?
Korff: No, for at some point I…
McGraw: I mean, you already had an MS, so you’d been to business school in some sense.
Korff: Yeah, I did that later on but they don’t teach you anything about management in business school as far as I could determine, at least the school I went to. It was more about building a business and understand the financials and strategy all that jazz. But actually managing people, no. Actually I stumbled upon—well, I had a good mentor at AOL for a while, just a really great sounding board. And then I stumbled upon an organization called Manager Tools, interestingly enough, also does a bunch of podcasts.
McGraw: Was that Doug Bigelow?
Korff: No, Doug Bigelow did run the organization at AOL but the mentor was Lavena. Doug was pretty ill when I started.
McGraw: He was up higher too, I guess.
Korff: Yeah, he was Lavena’s boss.
McGraw: Yeah, he was very good. His legacy is still around in a bunch of people that I know in the field. It’s very interesting.
Korff: He really had a good attitude. I remember I talked to him early on at my role there as I was moving to a management role. He said, “You know, whatever you love about the technical work, people are going to tell you you need to let that go. I’m going to tell you don’t. Hold on to it. Make them pry it out of your hands or your dead cold fingers.”
McGraw: Wait a minute. Now you’re getting me confused about the Superman Syndrome. What are you saying?
Korff: I never promised I would make any sense or be coherent.
McGraw: Well, we better quit then. It’s been about half an hour. So, last question, which has nothing to do with anything. So you’ve had a huge fish tank for many years. What’s your favorite creature and how big is your current tank?
Korff: Oh, man. The tank, which is the easier question, is 300 gallons.
McGraw: Holy cow.
Korff: That’s the main display tank. Then you’ve got the filtration system underneath it, which is probably another 100 gallons, and the refugium, which keeps an environment without predation, which is probably another 20 gallons. And then we get the back room, where I’ve got a water reservoir. All told, there’s a bunch of water involved.
McGraw: But you just tell your wife it’s 300 gallons?
McGraw: Is that how that works?
Korff: That’s pretty much how that works.
McGraw: Okay. What’s your favorite creature in the pile?
Korff: Generally speaking, I think my favorite set of creatures are corals. I’m not in it for the fish. The corals I find way more fascinating. They have this weird symbiotic relationship with algae that live inside their bodies and they’re kind of immortal. They just keep growing and splitting and growing and you’ve got to work to kill them.
McGraw: Do you dive?
Korff: I don’t do scuba diving. I desperately want to find time to go do that.
McGraw: Oh, you’ve got to. I just got certified this spring because my son Eli has been diving for five years and he finally just dragged me along. His mother is also certified. And wow is it cool. Definitely do it.
Korff: Yeah, I saw you went on a trip and then you got that certification a little while ago and I was envious, without a doubt.
McGraw: Well, just do it.
Korff: Just do it. I’m very busy.
McGraw: Dude, you work for Korff Consulting. Give yourself some sort of project in, you know, Saint Croix.
Korff: Yeah, I totally should do that.
McGraw: Well, thanks Yanek. It’s been interesting talking to you. I appreciate your insights.
Korff: Yeah, you bet.
McGraw: This has been a Silver Bullet Security Podcast with Gary McGraw. Silver Bullet is co-sponsored by Cigital and IEEE Security and Privacy Magazine, and syndicated by Search Security. The May/June issue of IEEE S&P focuses on the economics of cyber security. It also features our interview with Jacob West in which we discuss the IEEE Center for Secure Design.
Check out the video we shot to celebrate episode 120; 10 years of the Silver Bullet Security Podcast, and it includes drones.