Gary McGraw: This is a Silver Bullet Security Podcast with Gary McGraw. I’m your host, Gary McGraw, vice president of security technology at Synopsys and author of “Software Security.” This podcast series is cosponsored by Synopsys and IEEE Security and Privacy Magazine. For more, see www.computer.org/security and www.synopsys.com/silverbullet. This is the 136th in a series of monthly interviews with security gurus, and I’m super pleased to have with me today Pavi Ramamurthy. Hi, Pavi.
Pavi Ramamurthy: Hello, Gary. How are you?
McGraw: Pavi Ramamurthy manages the security ecosystem at LinkedIn. The security ecosystem team holds much of the responsibility for software security at the firm, including training, awareness, bug herding, application vulnerability response, program management, and security positioning for partners and customers. Pavi has over 20 years of experience in software engineering and development, coupled with 10 years of hands-on security experience. Pavi has also worked in various capacities at VMware, Determina, Vitria Technologies, and 3Com. Pavi holds an M.S. in computer engineering from Santa Clara University, and she lives in Silicon Valley with her family. Thanks for joining us today.
Ramamurthy: Pleasure to be here. Thank you for having me.
McGraw: It’s great to have you. You spent many years in professional software development building stuff, products. How long were you a developer?
Ramamurthy: Let’s see. Three years here…about 15 years.
Ramamurthy: Writing software development.
McGraw: Doing hands-on software development. So back…
Ramamurthy: Doing hands-on development, yeah.
McGraw: Back in those days, what did you know about computer security?
Ramamurthy: Not very much, I would say, up until 12 years ago, when I worked for Determina, which had this cool technology called memory firewall, and I was charged with the task of writing a management application for that. So that was my first interaction with security.
McGraw: I got you, and that was more about kind of making security work. What about software security? When did you first…when did that first occur to you as a person in development?
Ramamurthy: So that was a shift in career when I joined VMware about 10 years ago. I wanted to have a change in career, and the security team—which was at that point part of continuing development, and they were…they had an SDL team, and I joined as a program manager for the SDL team. That was my first introduction to security, thinking about software development, secure software development life cycle. And that’s when I first met you as well. That was my first interaction was with you, in fact.
McGraw: I do remember that actually, at VMware. So the point I’m trying to get across asking you these questions is there are a lot of developers that are extremely experienced and very good, and they could be good software security people, but often they don’t really know much about software security in the beginning, although I think that’s changing. So what do you think developers today should know about software security?
Ramamurthy: So definitely software security is evolving, not just from the security mind-set of security professionals but definitely from a software development point of view too, right? So I feel that it has definitely evolved, and more so that a lot of companies have a lot of security practices that are in the coding phase, I think Cigital had an IDE open solution, but then that gives them the knowledge and the awareness and the tools that are needed to, as they develop, know what kind of bugs are they looking for. “Oh, this is a cross-site scripting,” “Oh, I’ve just introduced a buffer overflow.” So it’s an awareness that is gradually given to them, over a period of time, I think that they are more cognizant of secure coding practices that they inherently try and put that in their code.
McGraw: Yeah, I think that’s especially true from a coding perspective. Like if your IDE is looking for particular bugs, it can give you a warning, “Don’t type that. That’s bad. We taught you that last week.” But there are other aspects of software security too, like security testing and risk analysis. In terms of that stuff, how much do you think the developers should know versus there should be some other people in the organization, like the software security group or the security ecosystem, enhancing those practices? I’m just trying to get a feel for how you think the balance should be.
Ramamurthy: It’s a combination of all of what you said, right? And what is the end objective? The end objective is that given a period of time, the software developer is equipped with all the skills that he or she needs to do secure coding, testing, implementation, design at a certain point in time and given the right set of tools, right? So how can we facilitate that? Every company has the Agile model or the Waterfall model, and they also have—most companies these days have a secure software development life cycle that is embedded as part of the release management process, right? So here at LinkedIn, we inject that security awareness at different points in the game, whether it’s Office Hours, having a Champions program, where we hand-select people to enable that security posture in them. We do have a software security group, which consists of pen testers and design reviewers, who will sit with them to explain security design principles to them.
So it’s awareness and training, and over time, you’re enabling the developer to think of those security concepts themselves, but it takes time.
McGraw: Yeah, and I think it takes a while for organizations to adopt that culturally. I mean, you guys have been doing it for a long time at LinkedIn. There are other people that are just starting. Firms have slightly different maturity in terms of getting these activities put into their SDLCs. And I use that with an “s” because everybody always thinks they have one but there are really 40.
So with regard to being part of the software security group—you mentioned that—do you think that nondevelopment people who really don’t know much about software can be effective members of the software security group?
Ramamurthy: Why not? I mean, absolutely, right? And it’s how you take that awareness to them. So to quote an example, we run the Security Champions program here at LinkedIn. We’ve successfully run that for a period of 2 years now. It started out to be mainly targeted at engineers, right? So it’s easier for them to understand the secure development life cycle. But over the last two programs—and our programs run for 6 months—over the last two programs we have included nontechnical people.
McGraw: Oh, cool.
Ramamurthy: Yeah, so you focus on an area or a skill set that is comfortable to them, right? So if they are working with teams to, let’s say, fix bugs and get good traction on bugs, whether it’s called quality or security or whatnot, we enable them with the right tools to be that evangelist that they can go and explain a security issue, even though they are nontechnical. And we tell them, “Hey, here are some courses. Here are some tools. Here is how the security group operates at LinkedIn.” And it’s all about enabling, in my mind. How do you enable? And it’s more of a personalized program, which is it’s targeted. No two programs in the Champions program that we run here are the same, right? Every single program with every single champion is very unique and tailored and personalized to his or her area of expertise.
McGraw: Yes, so what you’re saying is there’s lots of stuff to do, and you’ve got…everybody can do something. So you don’t just have to be technical person.
Ramamurthy: Absolutely, yeah. You don’t need to be a software developer. You can be a salesperson, and I can give you the right security tools to be able to respond intelligently to a customer who is asking a security question. I can train you to respond to information security questionnaires that the customer sends to the sales contacts. And I can give the same kind of training to our procurement and strategic sourcing and our legal teams to give them the right sort of material—what to watch out for in contracts, what to watch out for in data protection agreements. So the list is like…it’s endless.
McGraw: Yeah, that’s interesting. I kind of thought since you grew up as a software developer, you might have a bias towards software development people.
Ramamurthy: No, I don’t, I mean, and I have LinkedIn to thank for that because I actually do this in my everyday line of work here, which is interact with sales, interact with marketing to put out security white papers, interact with procurement, legal, and strategic sourcing to review contracts, interact with our suppliers. The LinkedIn security program is very unique in that way that we have a high touchpoint with almost all of the teams here.
McGraw: Yeah, that’s interesting. So tell me what got you interested in security when you decided to make that career transition. Why did you do that?
Ramamurthy: I wasn’t really looking for it. I mean, it just happened that way. And I wasn’t really looking to move into security, but the day I did, it was very interesting as to…and rewarding as well when you see how much of a high impact you can have on multiple organizations through, I mean…
McGraw: Yeah, no doubt.
Ramamurthy: Throughout the company.
McGraw: Well, in some sense you got started pretty early, because 10 years ago there were not really that many people (in 2007) practicing software security every day.
Ramamurthy: No, no, yeah.
McGraw: So that’s…you’ve probably seen…
Ramamurthy: I even remember that, yeah, the Gartner Magic Quadrant and all that. I just still remember that. And no, it was we were learning every day as we were evaluating new products. Like are we looking at enterprise software, it was more of a let’s have a design review and a pen test, right? I mean, it was very focused on the SDLC and very product oriented. But the awareness, the training, the contracts, the sales—I mean, we are adding more and more organizations within the company to the security portfolio, right? So that’s just multiple verticals that you can target.
McGraw: Do you think we’ve been making progress in the field of software security in the last 10 years?
Ramamurthy: Yes, and we are also discovering new issues and new threats as we go, but I do believe that we’ve had a lot of progress in writing automated tools. Solutions to do most of this work, so you can really focus on the very intrinsic, very nuanced security issues.
McGraw: So one of the things that we’ve noticed in BSIMM8’s results—which we haven’t released yet, they’re getting released in the fall sometime—is real growth in the field in terms of lots more firms are getting involved, and some firms are just getting started—lots of firms are getting started—but there’s a little bit of slippage in the overall maturity of the approach because there are lots of newbs coming into the field. Do you see that too in your interaction with your partners and your customers, that same phenomenon?
Ramamurthy: When you say newbs, you’re talking about companies? You’re talking about security professionals?
McGraw: Well, both. I mean, people that are new to software security as professionals and also companies that are just taking it on for the first time.
Ramamurthy: Mm-hmm. Yeah, I mean, we haven’t seen much, because our organization, the house security organization here at LinkedIn, has very experienced people from the security industry who work here. But having said that, the Champions program is looking at kind of newbs, right? Or people with not as much exposure to security but who have a passion towards security.
McGraw: Sure. I wasn’t really thinking about your own employees. I was really thinking more about the employees of partners and the employees of companies that are your customers and even the users of LinkedIn. There are some people that are just getting started in software security, and some of your partner firms might…you might be teaching them. “Here’s what we expect out of you.” And it seems like as the field grows there, we get more of that. That’s just what I’m thinking.
Ramamurthy: Yeah, yeah, and also it depends on really, what is the organization’s approach to security? How does the…I mean, is it enough if your product is secure? Is it enough if your third-party data is secure? But then how about a very simple thing as a contract? How about a simple thing as a breach notification? Who thinks about those things, right? So how do you educate your partners? And then partners, in turn, would turn around and say, “You know what? LinkedIn or some other company X gave me this set of data security requirements. I feel that they are great. I’m going to adopt that strategy, and in turn, I’m going to make that a requirement towards my suppliers or my customers.” Right? So it’s kind of a ripple effect that trickles down, but there’s a lot of knowledge that you can pull from various companies, at least here in the Bay Area, and how they…and it’s a very inclusive community where we can be somewhere, we can talk about and exchange ideas. Like the Office Hours was something that a lot of companies showed interest in. I mean, how do we exchange ideas? How do we have an inclusive community to say, “Hey, this is something that I’m doing, and this is maybe something that you can consider adopting as well”?
McGraw: Yeah. I thought we might talk a little bit about Office Hours. So what role do open Office Hours that the SSG holds play in LinkedIn’s approach to training and to awareness among the whole population over there?
Ramamurthy: Our Office Hours, broadly speaking, you can think of it as product security Office Hours, which are Office Hours that we have a couple of times a week that’s dedicated to any product team that has a specific question on a security feature, or even they say that “here’s a new feature that I’m implementing.” The best thing about the product security Office Hours is we have representation from house security, we have representation from our legal department and trust and safety, so we are sharing the same message to all the audience, and it’s consistent. We make a determination on what is the security impact if any, what needs to be done, is there a legal concern—that kind of stuff. So pretty straightforward, and we’ve been running this for over 3 years now, so teams are very well-aware of the process, and we do regular socialization of our Office Hour process.
McGraw: And they…are they using it? People come all the time to see what’s up? To get help?
Ramamurthy: You won’t believe it if I tell you that our Office Hours, our standard Office Hours, are always overbooked.
Ramamurthy: So we have SLAs. We have internal SLAs. We monitor our Office Hours, and very simply, Gary, it’s a no-brainer, right? And because it’s very simple. It’s a calendar slot that they sign up for. So we have our own SLAs. We monitor it on a daily basis. We open up more Office Hours. No team at LinkedIn would have to wait more than 2 weeks for an Office Hour session. And if it is urgent, they send us a quick note, and we do a one-off.
McGraw: That’s really cool.
Ramamurthy: That’s the product side of things. Yeah, that’s the product side of things.
McGraw: That’s a great way to interact because it’s sort of a pull model instead of a push model. People are often talking about you have to sit in training; you got to click through this stuff or take this CBT. But being able to go and talk to a live expert is just way better than that.
Ramamurthy: Yeah, and most often times, it’s a quick question that they immediately get an answer. On the other hand, we have business engagement Office Hours, which is from a supplier’s side, right? A very simple example would be I want to bring in a transportation system to transport employees from LinkedIn to the train station. I’m just giving you a very basic example. And then it comes to Office Hours, and we ask the question, so what kind of data do you need, right?
Ramamurthy: And they’ll say, “Oh, you’ll have to badge in,” right? So we use the employee badge, that kind of a thing. So, and then we ask them the question, what data is it the employee ID that is transferred? Do you just need something to say that they work at LinkedIn? So it seems that, hey, it’s just a transportation system. What’s the big deal? But we are highly protective and very possessive of our data, and we are very careful as to what needs to be shared. What is the bare minimum that needs to be shared in order to get the solution to work?
McGraw: Sure. You follow the principle of least privilege, from Saltzer and Schroeder.
Ramamurthy: Least privilege, right, yeah, yeah. And from the business engagement Office Hours, we look at contracts, we look at suppliers, we look at all the data security clauses in the contract. So it’s a high touchpoint with a small team. We have been very, very effective in having the Office Hours.
McGraw: We’ll be right back after this message.
If you like what you’re hearing on Silver Bullet, make sure to check out my other projects on garymcgraw.com. There, you can find writings, videos, and even original music.
So let me ask you about rapid development. I know you guys do Agile and CI/CD over there. And I’m wondering what you think the impact has been of more rapid development methodologies on software security in general. And do you think that they can both live in the same environment and everything is fine and things are moving along well?
Ramamurthy: For the…I would say about 60–70% of it is working well, right? So when you are doing Agile development, when you are doing rapid CI/CD, when do you…especially from a security perspective, when do you need to engage with security, right? If you have all the tools in place and you have all the automated testing and all you have to do is run those tools, get a security score, and you’re good to go, and you just present that score and say, “Hey, look I’m over 80%. Can I release?” That would be the ideal situation, right? Especially if it’s a tool that’s written in-house tailored to the proprietary framework. Oftentimes, that’s not the case. That is definitely not the case. So that’s where I think something like Office Hours would be helpful, right? “Hey, it’s a quick question. I’m doing a quick feature. Can I release? Does it need any security review?”
McGraw: I have another question that harkens back to the beginning of our conversation, and that is, we can build these tools, and we can put them in the CI/CD that check the code, and we can even check own proprietary frameworks, and that can be automated, and it can be fairly quick even though you got to fix the bugs. But what happens to design review and threat modeling? I mean, to me that’s the big challenge in CI/CD everywhere. It’s like, what do we do about review of the design? Because we can’t do that in an automated way that I’ve found yet.
Ramamurthy: We can’t, and I can go back and look at the number, but here we do a staggering number of design reviews and threat models in our rapid development framework.
McGraw: I see. So you just put a slot for that, but people have to do it?
Ramamurthy: Well, no, we have gates in place. So the way it works is, if you want to release a feature, you need to file what we call a ramp ticket, and we are monitoring that ramp ticket on a daily basis. And that ramp ticket serves the purpose of informing the sales and marketing team when that feature is going to be ramped. So we just injected ourselves into that, and we immediately go and look at the feature and we say, “Well, this is not approved for production until you’ve had a chance to come to security review.”
McGraw: I got it. So there’s kind of filter that you’re applying in the process.
McGraw: So I want to talk about security testing a little bit too, and I’m wondering if you’re reviewing a new feature and you do some design review of that, how do you link that in with security testing, and do you think that QA can be led to do effective risk-based testing?
Ramamurthy: Eventually, yes. We still need to do some more work on that on that front.
McGraw: As a field, absolutely. Yeah.
Ramamurthy: Again, as a field, and even here I think maybe we are in the process of putting in some tools that QA can absolutely leverage and kind of sort of come to a scorecard that is acceptable, right? But to my earlier point about ramp tickets and all of that, at LinkedIn, every feature is pretty much like a story, right? You can trace it back all the way to a ramp ticket, which in turn is linked to an Office Hour, which in turn is linked to a design review ticket. And the design review would specify if a pen test is required, in which case that links to a pen test ticket. And then when the pen test finds a bug, there is a bug that’s filed. So you started a bug, you can go all the way back to when the feature request was logged.
McGraw: Yeah, that’s interesting.
Ramamurthy: So we have that system, yeah. We have that kind of a bread crumb that we have throughout. But still, there is a lot that can…we also have some automated tools in place that will catch cross-site scripting issues and stuff like that. But I still feel that there is no closure or there is no point where we say, “QA, you have all the tools that are needed for you to be self-sufficient.” In terms of QA security testing, there is still…we are still not there yet as an industry.
McGraw: Yeah, I think that’s right. Who do you think should do penetration testing? In your view, who are the people best equipped to do that?
Ramamurthy: It’s definitely a skilled pen testing team, whether it’s in-house or external, because these are specialized skill sets; these are specialized tools that need to be installed. But I also feel that we can, and that’s the Champions program, which gives them that kind of experience and knowledge. Over a period of time, I most certainly believe that our security QA team will be able to do security testing, but I would…I’m still, right now, I’m still in the mentality of trust but verify. I’m very happy you did the QA testing and you did negative testing and you did all these security test cases and they’ve all passed, but I would like to verify at this point. But I’m sure that that position will change as we get more and more mature and we add more use cases, more tools, and that…
McGraw: I hope so. And I think you’re pretty optimistic on that one. I’ve been waiting for that to happen for 15 years.
Ramamurthy: I know, we’ve been waiting for it, I’ve been waiting for it, but that’s the…I mean, that’s how you become scalable as a company. That’s how you increase the talent. And all of a sudden, I’m a QA engineer who cannot just do functional testing but I can do security testing as well. My resume just became so attractive, right?
McGraw: Yep. So the last question I have is completely different than all the technical stuff we’ve talked about and it’s a little bit personal in nature, I suppose. You struck a really successful balance between having a career at a very high-profile company where you’re doing a lot of stuff, and also raising kids, who turned out to be really very nice young adults. So how’d you do it?
Ramamurthy: I mean, as you know, my daughter goes to USC, and my son is just about ready to go to Carnegie Mellon later in the fall.
McGraw: Oh, congratulations on that. I hadn’t heard that one.
Ramamurthy: Oh, yeah, you probably didn’t know that, but thank you for that. It hasn’t been easy, right? But now it’s easy. For the last 6, 7 years, it has been easy, but it wasn’t easy all the time, right?
Ramamurthy: But I can tell you—and I give this advice to all my women friends and colleagues—which is don’t give up, because it’s worth it. It’s not just worth it from a career point of view; it’s worth it from even a personal point of view. Because when the kids see their parent being able to handle multiple…juggle multiple things in life, it gives them an enormous boost in confidence, and you serve as a role model to the kids. And I have heard this firsthand from my kids. So I owe it to them in fact, and they encourage me.
McGraw: Oh, that’s really cool.
Ramamurthy: I mean, yeah, I, in fact, told my daughter that, “Hey, I’m doing this interview with Gary, and I have no clue as to what questions he’s going to ask me.” And to that, she texted me back, saying that, “Hey, you can lecture us with no prep at all, and you do such a great job of that.”
McGraw: Just treat me like one of your kids. I don’t know about that, Pavi.
Ramamurthy: Yeah, I don’t know about that either, but that was…that definitely made me feel better, so.
McGraw: Yeah, but I mean it’s really interesting because it’s such a conundrum. As a professional woman, it seems like that’s a very hard thing to do, and you’ve done an exceptionally good job of that. It’s nice that you…
Ramamurthy: It’s also the support that you have, right? What support do you have from your colleagues, from your boss. That definitely goes a long way. But then there is also perseverance in my mind, and you know what? End of the day, it’s really not rocket science; it’s how you do effective time management. It’s how effective you are. If you’re at work and if you can get your work done. I’m sure everyone can get it done. I mean, there are always the exceptions when the kids fall sick and the husband and the wife look at each other, saying, “Who’s going to take the day off?”
McGraw: Yep. Those days are over now. Now they’re in college, and you’re just like, “Have some chicken soup, and call us in next semester.”
Ramamurthy: Yeah, yeah.
McGraw: Well, thanks for sharing that. So last question: What’s your favorite summer read? I’ve just been…one of my favorites this summer is a book called “A Doubter’s Almanac” by this person, Ethan Canin, which I really like. But have you come across a book this summer that you recommend?
Ramamurthy: I have been…I actually have been reading…I mean, I’m a very avid book reader. My latest two books that I just finished over the weekend was “The Twentieth Wife” and “The Feast of Roses,” both by an Indian author called Indu Sundaresan. It’s about the emperors from India and one of the Muslim emperors from India and how they took…how this king, Jahangir, took the 20th wife and how the 20th wife showed immense confidence and courage in all the nation’s affairs and had a seat at the legal table and all of that. So it has been very interesting.
McGraw: Wow. Do you always read…you like to read history?
Ramamurthy: I read anything. You give me a comic book, I will read a comic book. I will read anything you give me, including cookbooks, directories. I will read anything.
McGraw: I think that’s a good answer. Well, thanks for your time, Pavi. This has been fun.
Ramamurthy: Thank you so much, Gary. It was pleasure being here.
McGraw: This has been a Silver Bullet Security Podcast with Gary McGraw. Silver Bullet is cosponsored by Synopsys and IEEE Security and Privacy Magazine and syndicated by Search Security.
The May/June issue of IEEE S&P Magazine includes our interview with Kate Pearce, who has a very unique perspective on sexism in security. Check it out. Show links, notes, and an online discussion can be found on the Silver Bullet web page at www.synopsys.com/silverbullet. This is Gary McGraw.