Until recently, there has not been real pressure to have supply chain software vendors attest to the validity of their wares. But with the introduction of software into automobiles, television sets, and medical devices, software integrity has taken on greater meaning. Many industries have specific hardware procurement requirements for parts introduced into their supply chains, but what about software?
Software today is assembled with up to 90% of the final software code coming from a combination of open source and third parties. Doesn’t it make sense to test and know what’s inside that code?
In this two-part Code Review podcast, I talked with Mike Ahmadi, director of critical systems security at Synopsys, and with George Wrenn, CSO and vice president cyber security for Schneider Electric, about the purpose of procurement language.
You can listen to the podcast on SoundCloud or read the transcript below.
In Part 1 we talked about unintended features in software, both malicious and nonmalicious, authenticated and nonauthenticated, and how to establish trust. In Part 2 we discuss CVEs, context, and certifications in the procurement process. We resume the conversation talking about human input parameters in the supply chain.
Wrenn: So the introduction of entropy and the human factor. The human factor introduces a certain amount of entropy into the system as well. I think the more you can minimize those human input parameters—and so in your case, the one failure mode to your case is when an organization does not enforce the supply chain contract. Or the supply chain vendor that’s giving you parts, your software, your libraries, whatever you are using, your API—we live in an API economy now. There are many cases where fuzzing an API is the equivalent of doing a supply chain test. So fuzzing APIs is sort of a new open the box and look at the assembly to make sure it meets the requirement. So in that supply chain API economy, fuzzing and automation work. But where the human part comes in and it falls apart is when management or the supplier doesn’t take the contractually obligated action.
So you can have the best system set up in the world, but if the people operating the system deviate from—we’ve seen that in aircraft crashes. We’ve seen that although the templates and the automation existed—like, for example, in one crash there was a cultural phenomenon of a younger pilot not—I think it was Malcolm Gladwell’s book—where the younger pilot, because of cultural reasons, the first officer didn’t challenge the captain and say, “Hey, we’re too low and slow here, OK? This is a problem.” What ended up happening was he ended up driving the plane into the ground. So there are cultural and other things that also come into play. The more you can remove those cultural, contextual issues, the more effective all of it, the entire system, is.
Ahmadi: You bring up probably the best and most valid point of all, is that we have tools but what the procurement language does it sets up a contractual obligation with penalties for not adhering to the contractual obligation. The argument I always make about security is we’re not dealing with a technology problem. We’re dealing with a policy problem. Ultimately, that’s what it comes down to. So being that we have a contractual obligation, I guess one thing I’d like to ask you, George, is the application of metrics. What do you think of things like CVSS scores, for example, or CWEs, ways to actually enumerate things that you can use as part of the audit process? What value does that bring to the contractual obligation based upon these tool outcomes?
Wrenn: Well, as a management scientist who spent more than four years at MIT Sloan and some time at Harvard Business School, it’s widely known that you cannot manage what you cannot measure. We know that. That’s a proven thing. So there has to be a measurement system for it. Now, that being said, there needs to be a measurement system that actually maps to the product process at hand. I think it was Sir Austin Bradford Hill that said in the late 1800s or early 1900s, he said, “Do not let the glitter of the tea table divert us from the inadequacy of the fare.” And what he’s saying is that just because the numbers look good and the statistics look good, if the statistics are telling you that that horse flew from one side of the English Channel to the other, no matter what those statistics say, they’re probably wrong. So (A) you have to have a measurement system, (B) you have to collect data, and you have make sure you’re collecting the right type of data. So the CVE systems—MITRE provides a very good set of data—they need to be the right data for the process that you’re currently monitoring and measuring contractually.
Ahmadi: Well, that’s where the notion of what I refer to as context comes into play. And that’s exactly what I mean. Let me give you an example. One of things that Codenomicon, which was acquired by Synopsys, was well-known for was the discovery of Heartbleed. If it’s on a server, it’s a horrible, horrible bug. But based upon the way the CVSS score is calculated, it only scored a 5 on a scale of 1–10. It didn’t even score critical, which is 7–10. However, if it’s a server-side-based attack, then it’s an 11 to those people. If it’s client side, it’s trivial. It doesn’t really matter. It really depends on the context of where the bug exists. And that’s why, using your analogies, regardless of what the numbers mean, if they tell you a horse can fly across the room, they’re probably wrong. You have to fully understand the context in which the numbers are being applied.
Wrenn: Correct. So in those kinds of cases where you talk about the Heartbleed and the anomaly there, the collaboration between the test cases and the scoring we say, I mean in those types of cases, the context would be to understand the proper context that the horse was on a sailing ship that went across the English Channel in half an hour, right, and that’s how it made it. So if you understand that context, then you understand. So you have to make sure that you are mapping to reality. I think that is what your point is about server side is that it has to map to reality. And reality changes. But you also have to continue to evolve that. IoT, I bring this topic up as well in a lot of people’s minds, IoT introduces the mother of all false secure supply chain issues. Because now you’ve got a bunch of distributed systems that are interacting that have various and or similar origins of manufacture. Being able to manage that supply chain is very difficult so again the idea that automation and measurement come into even sharper focus.
Ahmadi: George, I wanted to ask you, the, what are you r thoughts then about in procurement what if someone were to come up to you and say as part of my evidence I’m going to show you a certification that we have. Does a certification have any bearing on this? DO you ascribe any value to certification?
Wrenn: Absolutely. And it does so n a couple of different levels, right? When I research certification, obviously I want a very stringent certification, but the reality is that when I hear Certification it demonstrates a physiological case to me that the organization is cognizant or aware of the issues and are aware that their customers upstream care about security, okay. So, for me, having a company that does a reasonable job certifying is a start, right. IT might not be a perfect job, but at least they are cognizant of the importance of certification framework, which tells me from a maturity perspective that they are further along than supplier B or C, who has no certification process whatsoever. So a, yes, I want certification. I want certifications that mean something. But given a choice of a company supplier that does a mediocre average okay certification vs no certification, I’ll pick the mediocre certification over no certification because it demonstrates to me that there’s a cultural understanding and knowledge and intelligence there that I care about my upstream customers and I’m willing to take action and effort and dedicate resources to secure what I make.
Ahmadi: I think that I agree with you, except I would add one thing. I would definitely start with procurement requirements, and what I would want to know is, how well does the certification map against my procurement requirements? Someone may claim that they are certifying something, but it may absolutely have nothing to do with your requirements in your procurement language. In that case, the certification really means nothing to me. However, if I’m aware that the certifying body has tested against a program that maps to my procurement requirements, then I would be willing to accept that as evidence, if you will, third-party evidence in this case that you are meeting the spirit of my procurement language.
Wrenn: Correct. It’s a lot easier to go into a company and say, You are already working toward, you already have a framework, some beginnings of some activity. It’s easier for me to go to them and say, “Here are my requirements that input into your activity,” and have them reasonably expect to execute. Versus going to a company that has no knowledge of certifications whatsoever and then putting it on their lap and saying, “Hey, here are my set of requirements that you have to certify to.” So there are a few things there. It gets back to maturity. Maturity is very important, how mature is the organization that I’m sourcing from. I prefer one that has some certification over none because I can mold them to my contractual requirements …
Ahmadi: So you’re saying that an organization that has been through certification process can serve as evidence that you are more open to a certification process that may be different than one they did before, in other words.
Wrenn: They’re going to be a more mature organization which may also indicate they are more flexible, more ahead of the curve. Now …
Ahmadi: I would agree with that.
Wrenn: … However, though, but there is, again, going back to UL, going back to TUV, CE, all the various different bodies that are out there to make sure that when you plug something into your house it doesn’t burn down, right. In those contexts, it is often good, we work, at Schneider Electric we work with NEMA, the national electronic manufacturers group, we work with groups like that that make sure that when someone develops a charger for a car that it has a NEMA socket, right, that when your car pulls up you’re going to be able to be inter operable. So there is a huge role for folks like you to play to have a third party view of this that we can all agree to and work through rather than having the common torque problem of nxn-1 complexity where everyone has a different set of measures and means to measure, that becomes chaos. Having standards like ISO 27001 and maybe the UL certification, whatever have you, those things …
Ahmadi: …UL 2900, 62443 …
Wrenn: …yeah, those things make it a common language, otherwise we have a tower of babel where each organization goes out with a different set of materials and it becomes impossible for a single vendor that’s a supplier to meet the requirements because they only have so much bandwidth, right? So we need to give them a dependable, accurate, and hopefully singular standard to meet so that they can have economies of scale and efficiency in meeting secure supply chain contractual requirements to your point, right? So that’s where I think Synopsys can play a role in helping to secure the supply chain contracts through measurement, monitoring, enforcement.
Ahmadi: Well, then again, that was essentially why we created that first procurement language document because organizations were saying essentially, contractually what can we hold our suppliers up to? You know? Show us what do we ask them, okay? And can we provide them with something that is meaningful and achievable that will give us a level of cyber assurance. One thing that’s really important to understand is procurement language is—let’s use the analogy with pollution, having some set requirements for testing and cyber assurance will mean that you will be able to move the bar for cyber security and get cyber security to something that is much more manageable than where we are today. Creating standards and requirements for pollution have not eliminated pollution; they’ve made pollution very manageable, so we don’t just have people keeling over from black lung disease as all the time, like happened a few hundred years ago. So you look out today the air is fairly clean, okay. It’s not spotless but it is much better than it was in the past. I think that’s all we are saying with security. If we can create a consistent procurement language and if we can create a consistent set of testable requirements that people test to we believe we can dramatically decrease the unmanageability in cyber security that we are facing today.
Wrenn: Now here’s the thing, though, and this gets back to my earlier point. We need to have that, but we need to understand that different people are from a maturity perspective in different places. So we need a gradient-like approach. We have a rule within Schneider, which is that we meet people wherever they are. So if group comes to me and they are the middle of coding a product, and maybe we didn’t get a chance to work with them on the requirements phase, then we’re not going to tell them to stop everything they are doing and throw everything out the window and let’s start all over again. We’re going to meet them where they are. And we’re going to say, “You’re doing coding, and start with static code analysis on the product, and then in version 1.1 we’ll go through the full requirement life cycle and all that. If it’s a legacy product, we go through this process.” Now that being said, there are other groups that are just starting out. And from them, from that beginning, we’re able to go through a much more confined, rigorous process where they go through training before they touch any requirements documents. The requirements documents are vetted to a T, and all the i’s are dotted and t’s are crossed, and then they go to the next stage of architecture review, so just like ISO 27034. One of things to be careful of is we have to have a gradient in there in the sense that we need a way to take people who are at a low to no maturity and bring them through to a higher maturity, because we can’t expect that they go from zero to a hundred in two seconds.
Ahmadi: Yes, I agree with that. And a matter of fact, this was a conversation I was having with someone earlier today. You have to get an organization on cyber security on what I call a process of continual improvement.
Ahmadi: And the problem is, and I’ll speak of the UL 2900 program, which is a much different assurance program, the UL Cyber assurance program, because unlike other security certification programs before. One in particular that I am aware of it was an all-or-none—you had to do everything on there, or you did absolutely not get any recognition for what you did. For some organizations, that is extremely costly and time-consuming and they simply say, “We’re simply not going to do it.”
Wrenn: But if you are making a jet engine, you may have to do that. But you are a much more mature organization.
Ahmadi: But UL 2900 said, “Look, if you just want to start with testing one device, and you only want to run certain tests based upon what you believe your risk assessment is and your threat model is, you can do that. And all we’re going to do is verify that you’ve run those tests and passed. And it doesn’t matter what level you are.” And I think that is a very good approach to take because what we are running into today is people are by and large so immature in the cyber security process today that they simply cannot possibly pass an extreme battery of tests.
Wrenn: So the other analogy is—analogies are important, and I fell sometimes a bit Kafkaesque in using them—it’s like taking your child out to play catch baseball for the first time. You don’t throw a fast ball at 85 miles an hour. You start with an underhand, gentle toss of the ball, to catch the ball from a foot or two away. Now, when your kid is in high school, he or she is playing on a baseball team, then yeah, you might decide you are going to throw the ball at 50 miles an hour from 20 yards away because they are playing at a varsity baseball level and that’s acceptable. So I like to think of it in terms of an actual evolution and, again, understanding where folks are and giving them the right certification at the right time for the right context of the application or the end product that’s going to be implemented.