Show 028 - An Interview with Bill Cheswick

July 15, 2008

On the 28th episode of The Silver Bullet Security Podcast, Gary interviews Bill Cheswick, a lead member of technical staff at AT&T Research and all around security guru. Bill has been working in computer security for over 35 years. He coined the term “proxy” in 1990 with reference to firewalls, and co-authored the book Firewalls and Internet Security which was used to train an entire generation of sys admins. Gary and Bill discuss whether we’re winning or losing the computer security war, how security threats have evolved from pimply-faced teenagers to organized crime, whether we should move security into “the cloud,” and whether re-naming “Christmas lights” to “solstice lights” would bypass NJ holiday decoration ordinances.

Listen to Podcast

More Information


Gary McGraw: The first edition of Firewalls and Internet Security appeared in Las Vegas [at Interop] in 1994, along with a first of a huge pile of commercial firewalls. Back then, there were fewer than 20,000 Web servers—only 4,000 or so at the beginning of 1994 and approximately 20,000 at the end. Are we winning or losing the computer security war?

Bill Cheswick: I’m a pragmatic kind of guy. The bottom answer at the moment is everything’s working. If you ask a room full of techies how many people are doing online banking, maybe 70 percent raise their hands. “How often do you go to Amazon or Barnes and Noble or other online stops and do shopping?” That stuff’s working really well. Get your news on the Net. Use Google to search for stuff. For most people most of the time, the Internet’s working great, so I would say in that sense, we’re doing fine. Now, of course, there’s this lurking possibility of giant meltdowns and unsafe software, and all this other stuff that us security weenies are especially aware of— that this might be a house of cards that will collapse some day. But at the moment, it seems to be fine.

McGraw: Some researchers claim that attackers have evolved from pimply-faced teenagers perpetrating virtual joy rides to organized criminals perpetrating serious fraud.

Cheswick: I am convinced that’s exactly true.

McGraw: Have you had experience dealing with both the pimply- faced people and the organized crime people?

Cheswick: I have friends in law enforcement and people who watch the dark side very closely, and I monitor what they’re telling me. It’s quite clear that though there are some pimply people out there, the people we really need to worry about are the experts.

McGraw: Do you think the experts are evolving technologies in ways that the pimply-faced kids can’t?

Cheswick: There’s no question that they look over the shoulder of the kids. These kids come up with new ideas; I’m not saying that they can’t [evolve technology]. But the pros write excellent code, they’ve got a budget and so on, and they’re making lots of money.

McGraw: Have you ever been a victim of virtual fraud yourself?

Cheswick: To the best of my knowledge, I have not.

McGraw: Congratulations.

Cheswick: Some of it is personal care, and some of it might be whether I’m on the radar. It’s not clear to me what the reasons are.

McGraw: In 1990, you coined the term “proxy” in relation to firewalls in the paper, “The Design of a Secure Internet Gateway,” which I read recently. Now everybody uses proxy firewalls, and we still have a huge security problem. Do firewalls suck?

Cheswick: Well, they don’t suck. They’re sort of middle-level security. In fact, the paper you described was a high assurance—well somewhat high assurance—beltand- suspenders sort of firewall.

McGraw: That was also back when you guys had one Internet connection for AT&T.

Cheswick: Actually, there were more than one into the company at the time. My design got adopted by the rest of the company, and at one point, there were six or seven of those structures in place. There were 130,000 machines inside. Eventually, you’re circling the wagons around Wyoming—and that, of course, is the problem with all big companies.

McGraw: What do you think about firewall technology?

Cheswick: It’s a useful tool. There was a brief period around 1997 when people thought, “Gosh, I have a firewall, I must be secure.” Well, that sort of naiveté went away quickly.

McGraw: I think there are still people who believe that.

Cheswick: I have not run into them recently, but I take your word for it. They are misinformed. It’s a useful tool—like a router or packet filter or software checking tools and all that other stuff. It’s helpful to limit who can access what parts of what networks.

McGraw: Do you think that proxy firewalls are being used in generally the same way you anticipated?

Cheswick: Yes. Obviously, the traffic that they’re carrying is a little more sophisticated. Port 80 back then was pretty much a quiet, empty place. Now, it is a festering swamp with all sorts of things that are pretending to be Web services, plus Web services themselves. The job of the application gateway, of the proxy, for that particular service is much more complicated than it used to be, but the idea is pretty much the same and the use has been pretty much the same as well.

McGraw: I’m a big proponent of software security. Are we making any progress in that?

Cheswick: I have not seen any. It’s been obvious for 40, 50 years that we don’t really know how to program very well. The security implications are that we don’t know how to do security very well. I hope that guys like you win. You know it’s my computer, it’s my hardware, it’s my software, I’ve got the home field advantage. I ought to be able to win this. These guys [attackers] are remote. They’re in Latvia or someplace; all they can do is send packets to me. In principle, I ought to be able to win this.

McGraw: You shouldn’t accept those Latvian packets.

Cheswick: Well, I don’t know; sometimes people from Latvia want to talk to me. But you get the idea. I would love to have this fixed somehow, but I think for a while, we’re going to be stuck with the engineering problem of trying to engineer secure or reliable systems with unreliable components.

McGraw: Right. And you think that progress has been very little at best?

Cheswick: There are things that are better, certainly compared to 1990. For example, we won the crypto wars. SSH [Secure Shell], though not perfect—and I sure wonder if there’s a zero day that the Chinese government knows about in it—is much better than any tool we had back in 1990, and is probably good enough. That’s better. SSL [Secure Sockets Layer] is a good thing; that’s a nice tool. There are software tools out there that I think are settling down to being reasonably useful. I mean that not in a scientific robust software engineering good job, but in the engineering sense of “Well, we haven’t had many problems with it lately.”

McGraw: Yes, so we can use them.

Cheswick: Like Apache 1.3., whatever it is. That’s pretty much sitting there just sort of solidifying. Though it’s too big for me to trust; I spent the last week trying to jail it, in fact. Well, I can’t trust these things. We have things like that and Postfix and some other tools that I think are showing themselves to be fairly robust. I’m looking forward to virtualization.

McGraw: Is virtualization a boon or a menace?

Cheswick: Virtualization, you know, sandboxes, can be things like chroot and other things in operating systems, and much better things than chroot. That gives you belts and suspenders, the belt being the security of the original application. This gives you a third thing. We don’t really have a name for belt, suspenders, and what? Low-hanging limbs? I don’t know what it would be.

McGraw: One of those glass walls that the mimes use.

Cheswick: That’s right. Virtual machines can give us another layer. I don’t want to oversell it because the interface between the hardware and the kernel was never designed as a secure interface. Therefore, there are hardware-like tricks you could play to fool the kernel. They always trust each other, so this is not the most logical place to draw a security border. On the other hand, it’s a different kind of border and it’s just one more thing for the bad guy to have to break out of.

McGraw: When I came to visit you at AT&T Research, you were doing many cool things that I’m not allowed to describe.

Cheswick: Well, that’s true.

McGraw: What’s it been like to go through the AT&T breakup and then start-up mode at Lumeta and then the reassembly of AT&T?

Cheswick: This gets so complicated because people think that AT&T is Bell Labs—but that’s part of Alcatel-Lucent, and unfortunately, seems to be a smoking ruin. The folks at AT&T Research, at Shannon Labs—many are leftover from the old Bell Labs in the trivestiture of 1996. A lot of them are old friends of mine, many of them are new people. AT&T is a terrific place. It’s 300 very cool people doing very cool things. It feels like a slice of Bell Labs to some extent; certainly, the intellectual environment is like that. It’s wonderful to be able to go down the hall and ask a world-class mathematician a math question, or consult with a world-class statistician about a stats problem. This is the sort of stuff that was great at Bell Labs. This is where I belong, in corporate research; this is really just wonderful.

McGraw: How about your stint in start-up mode?

Cheswick: I loved the start-up mode. It was gratifying to take a research project and make it into something that people were willing to pay money for. I know it’s a bunch of shell scripts and cron jobs and C programs, but the guys who are buying it are paying 400 grand based on this sort of stuff and liking it. That’s a very cool thing. I think it’s also helpful for researchers to get out in the real world, see what the business is like, what business cases are like, go listen at venture capitalist conferences, and see what people actually have spent money on. This has been an invaluable experience for me. Now that I’m back inside AT&T, I’m more likely to hear an idea and say, “You know, there’s a business there.”

McGraw: It informs your tech transfer capability. That’s always the hardest problem in a research lab.

Cheswick: Well, it’s partly informing and then partly getting a large company to do something about it. That is also a problem. There have been a bunch of ideas that have come up. In fact, I’ve been cranking out patent applications, to my surprise—just sort of getting into this creative mode.

McGraw: I want to ask you two questions that I asked Ed Amoroso in a previous episode [See the March/April 2008 issue and www.]. What’s your opinion of moving security into the cloud? The idea being that it’s rather silly for each individual customer to invest so much energy on the end-point filters, like firewalls and IDS systems.

Cheswick: Of course, that was exactly the idea of a firewall in the first place—if you think of an Internet with no firewall, putting a firewall somewhere out there is in the cloud. I think it makes sense for some things. In fact, I tried to get AT&T into that business in 1991; that was a little ahead of its time. It’s now a billion-dollar business.

McGraw: Right. Yes, and people are arguing over who invented it. Not that we’d be speaking to one of those guys or anything.

Cheswick: Well, I’m not sure about inventing. It just occurred to me that “Hey, we got these firewalls and they’re pretty secure. Gosh, AT&T could sell safe access to the Internet. Wouldn’t this be a great idea?” That idea was not falling on fertile grounds in 1991 in AT&T, where they were still in love with the copper mine in the sky, the phone system. Frankly, that would’ve been too early to do a start-up with that, as well. It was an obvious idea. I think it’s a reasonable idea for some people to do that. What I’d much rather is that you succeed, and that my machine is just so armored that I can skinny dip on the Internet and not have to worry about this stuff.

McGraw: The other question is how do you personally feel about privacy, customer privacy, and the way they trade off against security?

Cheswick: I do not speak for any version of AT&T on this, but this has been an issue with the phone company—in any description you want to call it—for the last hundred years. On one hand, you have to run the network. That means you sometimes have to monitor phone quality and circuit stuff, and that means you have the ability to listen in on anything. You need that to run the network and you need that to run an IP network, a big ISP. On the other hand, there are privacy issues. The company obviously has a strong need to have customers trust us that we will do right thing. From what I can see from the inside, we do the right thing. We try very hard to keep this data extremely well protected—as near as I can tell. I haven’t run a personal audit on it, but the bits I’ve seen, it isn’t like I can just shuffle down to the end of the hall and go tap somebody’s telephone call. These things are studied for statistical purposes. When law enforcement comes along, well, the company has to respond. We just had a law passed that indemnified the phone companies for this, and I think that was a good thing.

McGraw: What about traffic analysis, the idea of who’s talking to who and when. Is that a privacy concern, too?

Cheswick: It is an incredibly valuable tool. I speculate that the spooks are much better at it than the rest of the world. There isn’t much literature on it out in the real world. I think it’s a rich place for research out here in the public world. Obviously, we have the databases inside AT&T that we could do this extensively, but, of course, for a research project how would you publish this? All the attempts I have seen at anonymizing data, whether it’s the Lawrence Berkeley Labs packet data, or some of the other stuff, have all backfired. This is something I wanted to do at Lumeta. Lumeta had seen, scanned in detail, perhaps 50 corporate intranets around the world. This is a view of networks that nobody else in the world has ever seen. Wouldn’t it be nice to sort of anonymize all this data and make it available for research, so that researchers could have some idea of the various mass and layouts of corporate networks? I could not think of a way to anonymyze this that didn’t look suspiciously like someone could figure out who the customer was.

McGraw: It’s a good open research question.

Cheswick: Yes. Well, we never did it, I just could not convince myself there was a safe way to do it. I think that we’re running into exactly the same problem with packet traces. I think this is a fundamental problem, an obstruction to research in the Internet and Internet security is getting access to packet data.

McGraw: I wanted to talk a little bit about the Internet mapping project that you did. If you were a bad guy and you had to abuse the mapping technology, how’d you do that?

Cheswick: Well, the best way to abuse it is to use it on someone else you’re going to attack. I think maybe you’re hinting at perhaps I could put up a fake network that would give false responses.

McGraw: I don’t know; I was just wondering if you’d thought through that.

Cheswick: The easiest thing is to just block it. You could offer up a phony network and have them come and scan the network and feed information. This gets into spy versus spy stuff. You could imagine that spiraling around a lot, and I’m sure it does to some extent. I think the technique is much more valuable on offense, and would counsel people that if you have low TTL [time to live] packets arriving at machines of interest, there’s someone showing unusual interest in what you’re doing.

McGraw: Right. Tell us an interesting story about an application of the mapping technology.

Cheswick: For corporations, just finding out where the corporate network goes was a big win. It was always interesting to look at the first map of a corporate network. These things would be colored with known and unknown, and that sort of stuff. Generally speaking, the network expert would look and say, “What’s that? Ha. We sold them two years ago. They were supposed to be disconnected.” Or, “Those guys in Oklahoma, I knew they were running an open router. Look. It’s right there.” You’d get stuff like that. It was often hard to tell what people wanted. Of course, we produced a report that produced answers to every question you might ask, as a big sea of Web pages. One of the reports was routing loops. Early in the Internet mapping project, we just threw away error conditions; we later realized that there’s gold in them there ICMP [Internet Control Message Protocol] packets. We’d find out routing loops and that sort of stuff. So, we made a report with routing loops, you know this, “We never got to the destination because your packets bounced around forever.”

McGraw: One would figure AT&T would like that sort of thing.

Cheswick: You certainly would.

McGraw: No, I mean keeping the loops around.

Cheswick: Well, that’s right. It keeps more traffic. There are people who saved money by just finding unused connections and said, “Oh gosh, we don’t need that T3,” or “That routing loop is using too much capacity.” What surprised me was I was told that when the network mapping project stuff was run on SIPRNet, the secret government network, the first report they looked at was routing loops. You could imagine that there might be some low-capacity top-secret lines that you’d want to make sure don’t get loaded too much. That’s pretty cool.

McGraw: That is pretty cool. Among your many interests, and your many interesting home automation projects, is a set of Christmas lights that are automatically controlled by a computer.

Cheswick: In a nontacky manner.

McGraw: Have you thought of uses for these lights during the summer?

Cheswick: No.

McGraw: How about possibly transitioning them to solstice lights instead of Christmas lights?

Cheswick: You could certainly do that.

McGraw: Do you think your neighbors would care?

Cheswick: I think they’d probably like it. I actually think there may be a local ordinance about how long you’re allowed to keep Christmas lights up.

McGraw: Well, then they wouldn’t be Christmas lights.

Cheswick: The other thing is I’m not sure those lights are really certified to work all year outside.

McGraw: I’m only thinking twice a year. We move up from just Christmastime to June and December 21st.

Cheswick: Well, yes, you could put up solstice lights. Of course, they wouldn’t be on very long, would they?

show 028 - Bill Cheswick