Software Integrity Blog


Podcast: Billy Rios on the good and the bad of Heartbleed, Part 1

Two years after its disclosure, the vulnerability in OpenSSL known as Heartbleed remains significant. There are valuable lessons still to be learned both about how the vulnerability was initially discovered and how the security community has responded over time.

This week my guest is Billy Rios, founder of WhiteScope, an embedded security company. In April of this year Synopsys asked Billy to research Heartbleed, two years later, and he found that despite a plethora of free tools and awareness, nearly 200,000 IP addresses remain vulnerable today. At InfoSec Europe 2016 this week in London, Billy will present a deeper understanding of his findings.

In this podcast, Billy discusses some of the good and the bad that came from the initial Heartbleed disclosure in April 2014. You can listen to the podcast on SoundCloud or read the transcript below.


Vamosi: What is Heartbleed and why is it significant?

Rios: Heartbleed was significant for a variety of reasons. The actual way the bug worked was interesting from a vulnerability standpoint. The bug was actually what a lot of vulnerability analytics called “beautiful” because the way that it worked. It presented a lot of characteristics that were kind of a 10 on a scale of 10. So if you are a bad guy and you want to exploit something, the way that it worked was actually pretty neat. So the heart of Heartbleed basically comes from a feature and this feature was enabled by default on a lot of different installations for SSL or OpenSSL installations, so it is very common software used to protect data in transit. Heartbleed took advantage of a feature called the “heartbeat” which was a way for you to interrogate a server to see what it supported or get some kind of basic diagnostic about who you were going to communicate with. And so the root cause of this bug was essentially an unchecked length where you pass a server a message and you would specify the length of the message and pass it to the server and the server would take that and basically issue a response. What it never really did, it never really validated the length of the message you were sending was the length of the real message, and if it were not, if the length were longer than the actual message that you were to send, it would actually respond with memory from the server, and the memory on the server could hold a variety of things. On thing already proven by researchers was that you could actually extract the private key for the SSL or TLS connections that ere established protecting people’s private information, their account information their user names and passwords and things like that, so … obviously a very, very bad thing. The reason people were, especially in the vulnerability world, the reason people were so enamored by this was because it was extremely difficult to detect. So determining whether someone had read the memory from your server is something that is hard, if not impossible to do after the fact if you don’t have security mechanisms in place before hand. Not like traditional man-in-the-middle malware where a malware file gets dropped on your server somewhere and is communicating with your C&C periodically for commands and the like. This is literally going to be really hard for you to tell that someone has used this against you in the past. A lot of folks actually brought that question up, with server use in the past, does someone else already know about this and used this, and the answer is we don’t know because there really is no good way to us to detect that and know. So a very interesting bug for a variety of reasons, technical and non-technical, but definitely a vary interesting case study.

Vamosi: You could initiate the heartbeat before authentication wa completely so you could smash and grab the encrypted information and that’s why you could leave no trace behind, is that correct.

Rios: Yes, certainly. I think when we look at vulnerabilities — if you work with vulnerabilities for a living — you usually classify vulnerabilities in one of two ways – you usually classify them as client side or server side, Heartbleed actually worked on both, which is pretty interesting. But the client side is where a lot of people are focusing their attention on. When we say client side bugs that means it is going to attack some bug that you have installed on your machine like your laptop, like a web browser, or maybe like iTunes, those things like that that you have installed on your laptop. A server side bug, however, is a server that is listening on the Internet, an FTP server, a Telnet server, an SSH server, and you have exploit to attack a server side service and those are very, very prized amongst in the vulnerability world. They are very hard to find because the exposure is just so great. So Heartbleed, the most impactful parts of Heartbleed, could be used to attack a server where you do not have to do something, you don’t have to send a phishing email, you don’t have to wait for someone to browse a web page; if the vulnerable server is out there, you can just steal stuff from it. Like you said, pre-authentication, before you authenticate you can do this from a Starbucks, from an airplane wi-fi, you can do this from anywhere. That’s what made it really difficult to detect. The other thing that made it difficult to detect is it is not like SSL is an uncommon service. People use SSL all the time. In fact, if you have a web page set up, SSL if one of the most common protocols you’ll see as far as the amount of traffic that is generated so sneaking in a request among the thousands or millions that come to you every day if you are a big bank or a well-known organization, finding that needle in the needle stack, even if you knew what you are looking for, and then trying to figure this out after the fact is going to be even harder.

Vamosi: How would an organization know if it was vulnerable to Heartbleed?

Rios: That’s a really good question. There’s a lot of controversy over Heartbleed and the ways some of things were done. For example there are a lot of people who criticize the fact that it had a logo and webpage and very generic description as for what it was and that sort of thing that got a lot of attention. But for me, to be honest, the amount of awareness this cybersecurity bug was unprecedented. So you had folks who had no play in cybersecurity who knew what Heartbleed was – it was on CNN. Many board members knew Heartbleed and when a bug gets that much attention, the folks in cybersecurity also pay attention. So there is no shortage of tools to help you detect whether or not you are vulnerable to Heartbleed. You can do this with your traditional vulnerability management tools – all of them have detection for Heartbleed. You could do this on the server side by doing binary analyses – I know a lot of different places offer binary analysis tools, like Codenomicon and Synopsys – you could do it that way, you could do it in a verity of different ways. If you were constrained because of your environment, you probably had a solution to figure out whether or not you were vulnerable to Heartbleed. The reason we had so many solutions was because it garnered so much attention. Pretty much every consulting firm that I know of basically created their own little Heartbleed check, certainly every vulnerable management service I know has robust checks for Heartbleed now. So they ways to check for Heartbleed or figure whether you are vulnerable to Heartbleed, there’s just so many different ways to check. And that is very unlike other vulnerabilities that don’t get as much attention. With other vulnerabilities you may be stuck with one way t check for it and if that doesn’t suit your environment, you probably have to create something custom in order to search for this vulnerability in your environment. With Heartbleed, not knowing how to check see whether you are vulnerable to Heartbleed, that is certainly not an excuse for this bug because there are so many different ways to check for this to see whether you are vulnerable to Heartbleed. It’s actually a really good thing.

Vamosi: Initially when it was disclosed in April 2014 there were estimates of about 600,000 vulnerable, I’m going to say, IP addresses, and that quickly went down to about half maybe six weeks, eight weeks later, and then you just ran a scan recently, you just looked at some data recently and we find that two years after that initial disclosure, with all of these tool available, there’s till something like 200,000 vulnerable out there …

Rios: That’s a really good point. When heart bleed was first announced like you said there was a lot of attention. Some people have actually scanned the entire Internet to see how many vulnerable servers there were. And I don’t think that has ever been done at this scale for any other bug, which made Heartbleed very unique and interesting from that standpoint. Heartbleed came out at a time where we really do have the ability to scan the entire IPv4 internet for certain things, right, so there are certainly a number of people who attempted to enumerate how many vulnerable instances of Heartbleed ere out there in the world and so what I wanted to do was come up with a very repeatable process where someone could easily verify what I did without having to set up their own Internet scanning engine, which isn’t that hard but still takes a little bit of effort . I used some data from it’s actually a web page and anyone can get this data. They canvas the entire Internet for you so you don’t have to do it yourself. They basically have a data repository of various artifacts that they collect. I used that data to basically see whether or not a particular server on the Internet on a given day was vulnerable to Heartbleed. We came up with about 200,000 servers still in April 2016 for the talk at Infosec Europe, where we’re going to have even more fresh data for that. It is a little alarming given the number of media outlets and the amount of attention that Heartbleed got considering that it is a server side vulnerable that anyone can export as long they can reach your server. I am a little surprised that there are still 200,000 servers out there that are vulnerable to Heartbleed. I don’t know what it says about our ability to patch and our ability to handle vulnerabilities at the Internet scale. Right, so, I’m certain that whoever owns these servers will have heard about Heartbleed or someone at their organization has probably heard about it. Like we said before there is certainly no shortage of tools that are available. Many of them are free — actually a lot of them are free — that you can use to check as to whether or not you are vulnerable to Heartbleed. The patch has been out for some time and so from a technical standpoint there’s no reason to be vulnerable on the internet to this anymore and so that’s pretty interesting. At Infosec Europe that’s kind of what we’re going to do, we’re going to deep dive into who’s vulnerable, like how many of these servers are in the financial sector. How many of these are in the academic sector, how many of these are in commercial real estate sector, things like that. What operating systems do, what these servers use, that could shed some insight in to why these servers are not patched. It’s a very interesting data set, to see so many servers still vulnerable even after all the attention that it got.

Vamosi: So where there other things that were good that came out of the Heartbleed disclosure.

Rios: I think it was people can argue that it was good and bad for a variety o things. Some of things that I thought was really good was there was a really great community response. If you were a small business, and had zero budget for cybersecurity, you could get remediation help to deal with your Heartbleed problem. And so whenever I see things like that, that’s pretty awesome. I don’t know of another industry that would do that. So if there was a widespread problem for your car, I don’t know of people who would freely just create things that would help you solve those problems. Right, so, it’s pretty amazing to see that the information security community come together, build tools, build capabilities, build services, build software for other people, give those tools, those software services away for free, just basically trying to get people to help people fix this problem in their organizations, that’s pretty awesome right? It’s pretty amazing and not many other communities can do it at the scale that information security does it. With that said, a lot people can view that as a model to get a lot o press and publicity and things like that so, Heartbleed has definitely become kind of the standard as to how you do certain types of vulnerability disclosure these days and some of them may or may not warrant that type of treatment. That’s different for every business. But sometimes it can make it hard for someone to get real work done, I think, because a certain bug has so much publicity and so much attention and now as a vulnerability manager you might find yourself explaining why that bug isn’t a big deal to your organization over and over and over again as people are constantly asking you so it’s a double edged sword. It’s just something we have to deal with in the formation security world.

Vamosi: You just touched upon it but are there things we could have done better looking in hindsight how things played out?

Rios: I always think that there are things you can do better. I don’t really want to turn this into “Hey, this is what researchers should have done,” or “this is what the vendors should have done,” I think we just have to realize that when it comes to software vulnerabilities, vulnerabilities in computer hardware and software, there are a lot of layers at stake. If you look at Heartbleed and the amount of industries that it touched, the response that it generated from various industries, we’re talking about banking organizations, we’re talking about organization that are running critical infrastructure that run our power and our water, things like that and so, and they all have different response mechanisms, they all have different capabilities, they all have different processes to do things, and so going back and saying hey, we should have done it this way, we just have to understand that, you know that two researchers basically changed the way the world sees vulnerability management. It’s pretty amazing. If you are a researcher and you discover the next Heartbleed, you have to understand that what you are doing could potentially touch the entire world. The entire world could be reacting to something that you’ve done. That you have discovered to a bug you are about to make public. And so it is a pretty awesome responsibility, there’s no recipe on how to do it 100 percent correct. And trust me, I’ve gone through disclosure many, many times, someone always after the fact will come up to me and say you really screwed this part up or you could have done this part better. And I agree with them and look for better ways to do things but it’s a very complex problem. It’s not as simple as putting up a webpage and a logo and just throwing it out there. I’m sure your organization was involved in a lot of the behind the scenes work that was done in order to make sure that the patch could made as widely as possible at the right time so people could get it, so people could easily update, to dealing with all these corner cases and all that sort of stuff, there’s a lot of work that goes into it. So, I actually applauded Google and Codenomicon (now Synopsys) when they went through this process. I thought they did it very, very well especially from a bug with the size and reach of Heartbleed so kudos to you all, definitely.

Vamosi: In part two of our conversation, Billy discusses some of the good and the bad around vulnerability disclosures in general.


More by this author