Software Integrity Blog


Introducing the 2018 CISO Report: A Q&A with Gary McGraw

Introducing the 2018 CISO Report: A Q&A with Gary McGraw

We recently sat down with Synopsys VP of security technology, Dr. Gary McGraw, to discuss his latest research effort. In addition to the annual Building Security In Maturity Model (BSIMM), Gary has set out to identify the ways in which CISOs approach their job role. The CISO project team, which included Sammy Migues and Dr. Brian Chess, interviewed 25 CISOs to identify approaches to the CISO role, characteristics of CISOs, and discriminators between types of CISOs and to establish a coherent model describing how CISOs organize and execute their work.

The firms the CISO research team chose to study include these:

ADP Aetna Allergan Bank of America
Cisco Citizens Bank Eli Lilly Facebook
Fannie Mae Goldman Sachs HSBC Human Longevity
JPMorgan Chase LifeLock Morningstar Starbucks
US Bank

Without further ado, let’s dive into the interview:

Q: What’s the CISO project all about?

A: The impetus really was to find out what CISOs do by gathering data directly from CISOs instead of having people self-report, or fill out a survey, or write their impressions about what CISOs ought to do without actually knowing any.

Just like in the early BSIMM days, there was a lot of confusion in the space about what CISOs do, and in fact, CISOs tend to approach the job pretty differently. We wanted to cut through all that haze and just do some data-driven analysis, and that’s why the CISO Report exists.

What kind of CISO are you? Get the report

Q: What did the data collection process look like?

A: I called up a few CISOs that are friends of mine and ended up having a telephone conversation of about 90 minutes with 12 of them. The reason for that was to build the framework for an in-depth, in-person interview of the BSIMM variety.

We wanted to make sure that when we flew out and did in-person interviews, we didn’t overlook huge swaths of territory. The first idea was “Let’s make a map of all the continents, and then let’s go fill in the continents in person.” So the hardest part of the work really—although it was the most fun—was flying around and doing the 25 in-person interviews that we ended up doing for this version of the report.

It was also amazing how forthcoming and eager to participate many of the world’s best CISOs were. I know when I called Craig Froelich at Bank of America, he was excited about the idea. It was something like a Monday, and he said, “Oh, well, can you come down Wednesday?” I’m like, “Yes!” And we got 90 minutes on his calendar, which is sort of unbelievable. And the same thing happened at JPMorgan Chase and Aetna.

So it turns out that because of the way we had done the BSIMM in its objective nature, these people knew that when we were gathering data and reporting on it we were going to be as scientific as possible, and they really liked that idea a lot.

Gathering the data was fun, but once we had that data in the first iteration, a year ago, it was still very difficult to find the right patterns. I was out in California having dinner with Brian Chess and talking to him about all this data that I had about CISOs and how it was way harder than the BSIMM to organize. The BSIMM was pretty easy because that was the field that we helped to invent. CISOs are a different story. So Brian said, “Wow, that sounds really cool. I’d love to help.” And I said, “All right, let’s all get together and do the exact same thing we did when we had the BSIMM data, only it’ll be harder.” We set a date, and I was hoping to have some pre-analysis done before Sammy and Brian arrived (those are the two co-authors).

Analysis went incredibly well. We had ideas of clustering the individuals and firms and then looking at major ways that the clusters had what we ended up calling “discriminators” in them. And then the four tribes just fell right out of the data. So once we hit on that idea in the first iteration, when it came time for the next 13 interviews to reach the 25 for the release that we’re putting out this time, it was much easier because all we had to do was verify that our first analysis actually was correct and that the data weren’t saying something different about what we should have done the first time. We basically verified our approach in our analysis with twice as much data for this release that we’re putting out.

Of the 25 firms in this study, many will allow us to use their names. There are just a few that would prefer not to be publicly named. But among that group, there are some stellar CISOs. Everybody who participated was incredibly helpful the whole time. They made time for us. They gave us feedback on our analysis. They’re excited about the work now. It’s just been a really cool project.

Q: In the beginning, what were you hoping CISOs would take away from this study, and has that aim evolved throughout the process?

A: I didn’t really set out to have a goal like that for this work. My goal as the scientist was to say, “I’d really like to know what CISOs do. I’d like to be able to describe it clearly and maybe talk about career progression and what types of people and organizations have the best security situations set up.” The idea was “Let’s go out and observe and describe the world.” In my experience—my whole career, actually—if you go out and you describe the world carefully, what happens is other people align with your description. And if you figure out a way to measure stuff, you can actually build a very valuable tool for analysis. So, believe it or not, that’s what we set out to do with the BSIMM too. In the beginning, the BSIMM was just, “Let’s describe the world carefully, clearly, and objectively.” The CISO Report is the same way.

What happens is when you get an analysis of yourself as the CISO, it’s hugely insightful because we’re comparing you to everybody else. We’re comparing you in an analysis that has a lot of data around it and is very well-organized. You can say, “My work in this role fits into this tribe and these are the 18 discriminators that apply to me.”  Point being, it helps you figure out whether it’s you as the CISO who needs to change your ways or the firm that you’re in. This is a complicated idea because CISOs are very high-level execs in usually pretty sizable organizations. So being in the tribe is not about an individual; it’s about an individual and the firm that they’re in. But once you have those data, it kind of opens your eyes up to the possibilities of what you can do.

Q: Can you tell us a little bit about the tribes?

A: That really is the meat of the report. The tribes are in reverse order, from the most basic tribe all the way to the most advanced tribe. We have Tribe 4, which is Security as a Cost Center. Tribe 3 is Security as Compliance. Tribe 2 is Security as Technology. And Tribe 1 is Security as an Enabler.

Just to put those all in perspective to each other, Tribes 1 and 2 are pretty closely aligned. Security as an Enabler and Security as Technology have to do with the background of the person in the role of CISO—either being a deep technologist who hasn’t quite let go of their technology roots or a hardcore technologist who’s transformed into a real business executive with gravitas. But those two tribes are very close together. If you think of them as over to the left, they’re pretty darn close. And there are some individuals that are right on the cusp between those two tribes. Then you go pretty far over to the right of those two tribes, and you get Security as Compliance where the organization is driven by compliance to such an extent that the CISO is sort of trapped in Compliance Land. Then, even further to the right of that, in a gigantic huge tribe, you have Security as a Cost Center. There, the CISO might not even be a CISO. They’ll be the person in charge of security, but they might just be a director, or five levels down from the CEO, or part of the IT staff and not really a person who is a senior executive, in many cases. That tribe is very big because the CISO role is relatively new, just like software security was relatively new a decade ago. It’s still being understood and being staffed.

Q: You briefly mentioned the framework that was created for this project. Can you tell us a bit more about what that looks like?

A: Sure. We built this framework that has three domains: Workforce, Governance, and Controls. Another way to put that is: People, Process, and Technology. So the framework with those three domains has nine sub-domains in it: Organization Structure, Management, Staff, Metrics, Budget, Projects, Framework, Vulnerability Management, and Vendors. That space is very large. I think I was surprised at how large that space was.

The framework allowed us to have these really in-depth discussions where we made sure we didn’t overlook part of the space, in the very same way that the software security framework makes sure that when we’re doing BSIMM interviews in person, we don’t forget to talk about some large aspect of software security.

I want to emphasize that what we do when we do both the BSIMM Report and the CISO Report as recurrent work, it’s a lot of very active listening. We’ll bring up a topic, and then we’ll listen and write furiously. We’ll make sure that the data we’re gathering is self-consistent and that we covered the whole space. What we don’t want to do is ask a series of checkbox questions. That’s not how the BSIMM or the CISO project works, and that’s what makes it interesting. When you have super-senior people that are really good and you’re guiding the conversation, you end up with excellent data. The data are everything in these research projects.

Q: What can a non-CISO take away from this project?

A: If you’re a security professional and your ultimate career goal is to become a CISO, this work is incredibly useful. You can find out what sorts of things you don’t know and need to know in order to make it to the CISO role. In some sense, I think that it’s very useful for people who aspire to be CISOs. I also think it’s very useful for people who try to provide services, tools, and technologies for CISOs—like what we do at Synopsys. We have to understand how CISOs think, what they really want, and what their problems really look like. The CISO Report does more for that than anything else in the CISO space that I’m aware of.

And when you know what someone is trying to accomplish, you’re in a much better position to help them get that accomplished properly. So I think that the data that we’ve gathered for the CISO project is going to be useful for just about everyone thinking about computer security and getting up over the technical arcana and into management—understanding the business approach, how you do risk management, what to measure, and all that stuff.

Q: What’s the ultimate takeaway for readers from the project?

A: The ultimate takeaway is to get your CISO involved in the CISO project. Have them reach out, request the report, and get involved. We’re still gathering data, and we’re growing the study. That’s what we’re going to do over the next year. And one of the real questions that we would like to answer, which we don’t have the answer to yet, is, can CISOs change their stripes? That is, if you’re a Tribe 3 or a Tribe 4 CISO, can you become a Tribe 2 or even a Tribe 1 CISO? Can you progress your career that way? And the theory is that you can. But we have to find out in practice whether that’s true or not.

What kind of CISO are you?

Download the report


More by this author