Software Integrity Blog


Fault Injection Podcast .002: What’s in your software?

Fault Injection Podcast .002: What's in your software?
Fault Injection is a podcast from Synopsys that digs into software quality and security issues. This week, hosts Robert Vamosi, CISSP and Security Strategist at Synopsys, and Chris Clark, Principal Security Engineer at Synopsys, go into detail about a new report: The State of Software Composition 2017.

You can always join the discussion by sending us an email at

Fault Injection, Episode .002: “What’s in your software?”

Or, read the transcript for Fault Injection, Episode .002: “What’s in your software” below:

Robert Vamosi: I’m Robert Vamosi, CISSP and Security Strategist at Synopsys.

Chris Clark: I’m Chris Clark, Principal Security Engineer here at Synopsys.

Robert: Welcome to Fault Injection, a podcast where we dig deep into the security issues of the day. We’re going to be continuing our series on cyber supply chain.

Chris: Not only are we going to carry that forward, we’re going to get to something kind of interesting. We touched on it in our last podcast where we were talking about software composition analysis. But there’s a new report that’s going to be coming out that you’ve been playing quite a role in.

Robert: Yeah. Synopsys has been working on a report, which we just released, and it focuses on the state of software composition analysis for 2017.

Chris: Are we talking about open source composition analysis? Are we talking about closed source? What’s…?

Robert: We are. These are applications that were uploaded to our Black Duck Binary Analysis Cloud instance. We have over a thousand uploads that we looked at, and we broke them apart into their constituent components.

What makes us a little different from some of the other software composition analysis reports is we actually have the data. We actually have data points that we can use. It’s all anonymized data, but nonetheless it’s fascinating.

Chris: I’m a big proponent of open source. I love open source tools. I’ve loved Linux for many, many years. Many of our listeners probably are quite familiar with open source. There’s been a lot of reports out there on what makes up open source. What are the vulnerabilities? What’s going to be different about this?

Robert: What do you mean different about this?

Chris: This report, say.

Robert: This report. We pull out the components that we see in these applications and we’re able to see the different versions that are used.

What’s really important to know here is, not only did you find that zlib was the most used component out of this group that we looked at for 2016, but there were versions of zlib that have CVEs associated with them, potential vulnerabilities that could be exploited.

If you’re using it in your software package, you probably want to update. You probably want to move beyond that. That’s where some of the problems have been, is that people are using really old versions of these components.

Chris: Part of that’s the development process, right?

Robert: Right.

Chris: When the product starts its initial development they look at, “OK, what do we want to use?” It may have been two years ago, and that was the latest library at the time. What we’re talking about is ensuring, throughout that developmental lifecycle, those libraries get updated?

Robert: Right. You touched upon something that I just want to expand on. Why even go there? Why even use some of these open source libraries?

When you’re creating an application, you want certain functionality, you want certain things to be there. If you’re not skilled in encryption, you don’t want to be rolling your own encryption, as they say.

You want to be able to grab OpenSSL. You want to be able to grab something that’s already been created, that’s been vetted and used. Fine. That creates another problem. We see a lot of companies banning the use of open source because in the past people have used these and have failed.

Heartbleed, for example. Two years, we had a bad implementation of the heartbeat feature in Heartbleed. That was not caught, and yet, Linus has said, “All eyes on open source are good. We’ll find these and fix these eventually.” Yet, here was an example where a lot of people looked at it, and it took a tool such as one that we had through Codenomicon to expose it.

Chris: That’s pretty typical. It’s this knee jerk reaction, right?

Robert: Right.

Chris: When we look at security, a lot of it is event driven. When we see an event like Heartbleed, or Poodle, or any of these other vulnerabilities that have popped, organizations react. “OK, no more open source. We’re going to write code.”

But historically, we’ve always found that organizations that write all of their own code try to make a new implementation of OpenSSL or some other protocol or library—you name it—tend to be inferior because we don’t have enough people looking at it. We don’t have the eyes on it.

In that vein, we also want to say that, why would you put an implementation out of Linux kernel 2.1 out there? You’re saying, whatever the latest and greatest is, that’s your focus and that’s what you should be driving your development cycle to.

Robert: Right. In the course of writing this report, it became clear that there were some components where we saw more older versions, more vulnerable versions, than others. We pulled them out and we called them the dangerous 10. That doesn’t mean that the developers of those open source components are bad and you should avoid using them in the future. Rather, we have seen more instances than not where there’s a vulnerable version in a application that was uploaded to our service.

Why? Why are you still using a three year old version of a component when there’s a latest and greatest out there? That type of awareness needs to be baked into the process of creating software.

Chris: Are we seeing organizations like Linux Foundation and Open Source Foundation be more proactive in this respect?

Robert: Yes. As you mentioned, a couple years ago, open source kind of got black eyes because there were faulty components, faulty implementations, whatever. They’ve really stepped it up in the last few years. They’ve really taken on the burden of making sure that their products are good. Whether or not it’s the thousands of eyes looking at it or internally testing it themselves, they’re making tremendous gains.

Again, it goes back to the people that are incorporating this into their applications. Where are they getting those versions of an open source component? Are they getting it from the actual organization? Is the provenance good where they’re getting the latest and greatest, or are they getting it from a friend of a friend, or is it legacy? “We always used this. Why not just add it in?”

Chris: “Why change it?” Right.

Robert: Why change it?

Chris: You mentioned the provenance of the source code. For some industries, provenance of code is irrelevant. “Just grab that library, get that product to market. Let’s get it done. Let’s get it out the door.” But provenance has proven to be more and more important, so are we able to be much more definitive in the actual provenance of code now?

Robert: In the sense of…?

Chris: When it’s built in the binaries, is it pretty clear to see that, “OK, this was the original library,” or it has somehow been tampered?

Robert: That’s funny you should say that. In looking at the data, we did get a number of unknowns. The unknowns means that something has been tweaked about it. It’s not necessarily the version that came from the open source organization itself.

Why are we seeing those variations? Don’t know. Should you be using them? I don’t know. Again, do you want to use a friend’s of a friend’s component, or do you want to use the one that directly comes from the…

Chris: There’s a big difference between an organization that has brought in an open source library and then forked their own tree for internal use. Versus, a tree that’s out there that’s been forked. We really don’t know why it was forked, what the what the differences are. It’s much more important to be able to determine that, especially when we’re bringing in third-party components from different organizations.

We talked about this on our last podcast, is that there’s a lot of third-party code out there. We don’t write very much code these days that’s internal to an organization. It’s important to understand what that provenance is and be able to track that over time.

Robert: You mentioned forking. There’s nothing wrong with forking, but you assume the risk. That becomes your code now. You’re developing it from the base that you got, but it’s your risk. Going forward with it, you need to be testing to make sure that you’re implementing it correctly and doing the right by it.

Chris: By doing that, you bring in a lot of risk. You have to maintain that now for the lifecycle of that particular product, or however long you’re using that library. It’s not just a one time instance because I wanted this one specific feature.

If that is the case, and you’re participating in this, maybe that feature’s important to others and you might want to roll that back to the public tree, make that open source. Challenges, licensing requirements, etc…but we diverge a little bit.

Robert: We are.

Chris: Let me ask you this question. You helped write this report. What were some interesting things that stood out, other than the top 10 that you mentioned?

Robert: We also looked at active CVEs against these components, as I mentioned. While 45% of them are within the last three years, and that’s to be understood that more recent CVEs are going to be more common, we had legacies going back to 1999. Why were we seeing CVEs from 1999?

Chris: [laughs]

Robert: That’s just a little crazy. Mind you, we only saw one instance of it. Still, it’s—

Chris: They exist.

Robert: Surprising to me. Again, it’s probably some legacy that was carried over, and no one thought to update that library for whatever reason.

You mentioned Poodle. You mentioned some of the other Ghost and Heartbleed are in that category. We call out some of the common name vulnerabilities and we looked at the instances there as well.

I’m happy to say that Stagefright has pretty much been mitigated. We saw very few instances of Stagefright, but we did see a lot of instances of Poodle. It’s toward the top of the list. Heartbleed was right squarely in the middle. We could spend a whole episode talking about why Heartbleed continues to be out in the wild.

Chris: That brings up something that I do find interesting. I heard you talking about this earlier, and you mentioned Heartbleed, which happened a number of years ago; a little over two years ago, yet we still see it. You mentioned something called code decay.

Robert: Yes.

Chris: Can you describe that for the audience?

Robert: In your software development lifecycle, you go through the testing. You make sure that when the code is released, you’re aware of the vulnerabilities that may be in that code at that time of release, and you’re comfortable with them. You’ve attested to the fact that, “I’m fine that these vulnerabilities are there. They can’t be exploited in the uses that we’re looking at.” Great.
You roll out the door. It is a secure piece of software, day one. Day two comes along, day three comes along, a CVE gets found against that code base. Day 10, day 20, day 30, more CVEs. As the code continues to age, it tends to get more vulnerable. Maybe now, a new CVE that was just discovered piggybacks with another CVE, and what you thought was a vulnerability that couldn’t be exploited, can be exploited.

If you’re not updating, if that’s not part of your software development lifecycle, if you’re not maintaining the software beyond the release state, that becomes a problem. That’s where I’m talking about code decay.

You not only have this graph, which is, “Here’s the baseline that I’ve established with my software upon release.” You’re having new CVEs going against that code over time, but you also have this other dip going on there where CVEs are piggybacking on the existing CVEs that you already knew about and making it exploitable.

Chris: That brings up an interesting challenge. We talk about software development lifecycle in some of our programs. As a developer, when I look at the code and I choose the libraries and I’ve developed my software, usually that gets chunked over the fence to another team.

What we’re also implying here is that other team has to have the tools to be able to monitor this product that’s been released and is out in the market. Without that information, without that continuous monitoring, you are going to be reactionary. You are going to be event driven.

There’s a critical component also, which is as we find these new vulnerabilities, as that new information becomes available, that has to get cycled back to the development team.

Robert: Right, there’s a feedback process.

Chris: A feedback loop that has to happen there. When we talk about that, that’s one of those things that is an educational process that we have to make sure we maintain focus on and we have policies and procedures in place to deal with that.

Robert: It needs to be the infrastructure for not only reporting and being aware of the vulnerabilities, but then passing it back to the developers who are responsible for it. Then, there’s a whole structural issue of how do you bake in this update process?

Microsoft has sat on updates for a while because they claim, for example, Internet Explorer has a lot of different vulnerabilities. They’d be patching it every single patch Tuesday. They save it up, and occasionally they’ll roll out.

What is your company’s policy around the application? How often do you want to be passing out? There are some downsides to updates. That is features and usability might change, and so users might grumble about, “I don’t like this new update,” or whatever. There’s also interactions that you have to test.

It’s a big undertaking. Just changing a library sometimes is more complicated than it sounds. We’re just saying you need to at least be aware that there’s old components, there’s old software libraries, and more to the point, there are vulnerabilities in there.

Chris: Everything that we’ve talked about, do you feel open source is getting better? Has it become more reliable?

Robert: I do. And there have been very public expressions of that. For example, the Mozilla Foundation has a fund that they will help open source organizations get the tools and get the testing to make sure that their products are more secure than they would’ve been before.

We work with a lot of these open source organizations to help them identify vulnerabilities and to address them early, again, improving the overall nature and quality of open source.

Chris: It is a challenge, right?

Robert: It is.

Chris: When we look at the overall lifecycle of software that’s out there, how long has zlib been around? How long has the Linux kernel been around? Those are just a few examples of many of the software tools that are available to us, both commercial and noncommercial.

When we look at SDLC processes, there’s many books, there’s many methodologies, but ultimately what we’re getting out of this is, maintain code. When you look at applications that are being developed, what’s going to be that long term cycle and how we manage that over time.

Robert: You mentioned the Linux kernel. We saw 170 distinct versions in the applications that were uploaded just in 2016. 170 versions that were called out. Of course, a majority of them were associated with CVEs.

There again, if you’re not looking at the latest and greatest, if you’re not grabbing the source code from its point of origin, you may not be getting the full benefit of using that.

Chris: One of the things that strikes me when we talk about this is many people say, “God, security just costs so much more to implement.” Just some of the things we talked about are rather simple to implement. Low cost, low impact, and really if it mitigates one event for an organization, most likely will pay for itself.

With that, we’re going to close out for today. We’ve really appreciated everybody joining us. We have an interesting episode that’s coming out next. You want to talk about that one real quick?

Robert: Sure. We have another report coming out with VDC. We partnered with them to discuss military, government, and aerospace industry. We’ll talk more about that the next time we meet.

Meanwhile, keep those comments flowing. Use the email address and send us your feedback. We’d love to hear from you. We’d love to incorporate your ideas into the show. As we’ve said at the very first episode, security is subjective, and we want to hear your perspective as well.

Chris: As always, thanks for joining us. We’ll see you at the next episode.


More by this author