Virtualization is a process, not a product
I see that the BlackHat Blogger’s Network has a topic of interest. I’ll oblige, especially since The Hoff is involved. I think it’s a good exercise, so I’ll have to thank Shimel for this idea.
You also won’t want to miss what I’ve said about virtualization four months ago in Hardware VM security: past and present. Today, I just want to talk about Hoff’s points and my experiences with modern virtualization.
Time is your most valued resource
Securing outdated “default-insecure” network devices, operating systems, and applications is wasted time that could be spent trying to convince management to replace or recode them.
It really does take about two or three days for one administrator to install CentOS, SELinux, Kickstart, and Apache/PHP. Really what else do you need? Similarly, I guess it takes about two to three months to install Windows Server 2003 or 2008, GPOAccelerator, RIS or Deployment Services, and IIS/ASP.NET. It takes maybe another month to implement the vendor/NSA/NIST hardening guidelines, CIS benchmarks, and OWASP material for either platform. If you can’t be bothered to download a few PDF’s, there’s always books like the Hacking Exposed series.
In the case of “if it’s not broke; don’t fix it”, the potential of a data breach due to an existing, known vulnerability (or an obvious state of software weakness) would count as “broken”.
Virtualization can help organizations move quickly
Virtualization is supposed to help with these efforts! Instead of running through a whole Kickstart or WDS install, all you have to do is copy a file and boot it. Better — this can be combined with iSCSI and shared disk filesystems such as OCFS2 or GFS. Better — iSCSI can be enabled with DHCP to quickly get to new services.
I’ve had Xen hardware virtualized guests that ran redundant services on the same machine, with a keepalived heartbeat running between them. Why? Availability was the primary goal. Availability is certainly a part of security.
When you want to replace everything and start from scratch — virtualization will help you get there much faster. In a mature infrastructure with mature process, any organization will see increases to availability (and therefore also the availability side of security) and mean-time-between-failure, and a decrease in mean-time-to-repair. Bootstrapping an infrastructure overnight is certainly possible. How is this not a good thing?
Virtualization has the same problems as the least privilege principle
The real least privilege principle is “if I have an account on the box, then I also have administrator/root”. Privilege escalation is almost always possible. The same is true with regards to virtualization. If you get access to a Xen/VMWare/VirtualIron/etc guest, then you can usually also eventually get access to the host OS (and therefore all other guests).
For those that don’t remember the Xen Multiple Vulnerabilities, read on up.
The nice thing about virtualization is when you take into account the previous concept. Sure, virtualization increases risk because it creates a situation of trust between the host and guest OSes. However, virtualization can also be instrumental in quickly installing and verifying SELinux RBAC or TE configurations, thus reducing that same risk to all OSes.
Also worth a mention here are concepts such as secure hypervisors (see my last article about sHype and links), Xen Access Controls, and Phantom.
Capacity planning is not as important as live (or dead) migration
I’ll concur with Hoff that security concepts hurt virtualization concepts with regards to capacity planning. However, I’m not certain that virtualization is all about capacity planning in the first place. If anything, it allows you to shut down redundant services when they are not needed (or turn them on only during peak times). Virtualization allows you to move OS guests around the network (btw: doing so securely would be a very good idea — I’ll have to write up a paper/concept on “MITM’ing Live Migrations”), so in this way you can fix performance problems quickly by just shuffling things around. Again, this helps availability because it helps mean-time-to-repair.
For those of you who missed what I said about live migrations in my last article, be sure to check it out — or just check out this link I had to an article on Live Migration of Xen Domains. The basics is that computers should be shut off when they aren’t in use, and with virtualization this could mean shutting down everything but the last box.
Inter-dependencies caused by virtualization
I’ve seen this, felt its effects, and dealt with the agonizing pain. Virtualization environments add complexity. If you don’t handle this complexity well normally, then stay far away from virtualization.
Example
In the “virtualization can help organizations move quickly” section above, I talked about using Xen guests with filesystems on a remote iSCSI shared storage device running a distributed filesystem such as OCFS2, I also indicated that you could use DHCP for the Xen guests to find their associated filesystems. Well, of course — I had problems with DNS, which was running in one of those Xen guests when everything came crashing down. After this event, the Xen guests couldn’t find their disks because they couldn’t find their DNS.
Hoff mentions ACL, VLAN, and other network-based dependencies. Of course! Don’t do this sort of stuff and you won’t get yourself in trouble. But this takes planning, risk management (of the non-security-only kind), and repair process analysis.
Documenting your environment and networks should already be in place before you move to a maturity step involving virtualization. Playing out “what-if” scenarios with a whiteboard can be done in the same way that Red-team exercises can be done. Sometimes being a security professional can also allow you to demonstrate your other resourceful risk expertise.
Virtualization isn’t a technology — it’s a transition
Allow me to bring all of this together.
Gartner has an excellent model for understanding IT/Operations called the Infrastructure Maturity Model. The [PDF] current version that I’m looking at right now has seven levels.
Gartner Infrastructure Maturity Model
- Basic
- Centralized
- Standardized
- Rationalized
- Virtualized
- Service-based
- Policy-based
Assume you’re in #1 (in the document above, it’s #0) if you don’t know otherwise. A lot of smart Enterprises are at least Standardized (#3 above), many more are Rationalized, and some have taken steps towards Virtualized.
I know that it is somewhat a theme of this blog (I also feel that it’s a theme of security and risk management in general), but basically what I’m trying to say is that process is more important than technology. Process needs to come first. If you think you can download Xen and integrate it into your production IT/Operations infrastructure while you’re stuck in phases 1-3 of the Infrastructure Maturity Model, then you need to take some ITIL classes. If you think buying the latest VMWare, VirtualIron, or Microsoft Virtual whatever is going to help you if you don’t do the documentation, monitoring, life cycle care, strict change management, refined repair processes, and enduring risk management — then you’re just plain wrong. Virtualization is process, not a product. Just like security.
Security and virtualization can complement each other, just like security and availability (or ability to change) can compliment each other. If you didn’t read Hoff’s post on The Challenge of Virtualization Security: Organizational and Operational, NOT Technical — please do so. What’s more is that Pete Lindstrom and Mike Rothman say fairly similar things to what Hoff and I are saying. I’d say that we’re all pretty fairly united as an industry on this topic, which is rare.
For those that want to read more on this topic, I suggest checking out this book on Virtualization Security from Michael T. Hoesing when it comes out in the next week or so.
I’m Interested in what you’re saying about least privilege and guest–>host compromise. Do you think that the same caveat applies to “bare metal” hypervisor based virtualization environments (eg, ESX server)?
There seems to be a lot of business interest in the idea of Virutal machine farms that will host guests in multiple logical network segments at different classification levels, and if it does turn out to be possible to compromise the virutalisation segregation in some/most/all cases then that business model would seem to be very broken.
@ Rory:
I don’t see why it wouldn’t apply to bare-metal solutions like ESX server. Worse, ESX server could contain backdoors that enable this functionality, ones that might be very difficult to detect. Look at Cisco IOS and other embedded OSes — same deal with privilege escalation and rootkits.
In 1998, I was part of team that tested and exposed problems with CatOS software, including testing of VLAN’s for security purposes. For a long time thereafter, it was thought that VLAN’s violated RFC 2196’s rules of separation.
Turns out that later I was one of the biggest proponents of using firewalls on a stick (or a firewall on the same switch, with different VLAN’s for the external and internal ports). This often confused people — why would I go through all the trouble of proving that something is insecure, later to only use it in that same way?
In some ways, these are questions that you are going to have to answer for yourself. With sHype and SELinux RBAC/TE, it might be the case that virtualization concepts still break the classification levels. However, so do the assurance models, if not taken into account (e.g. buffer overflow on a remote service or an open DAC policy on the memory or filesystem).
Many of these answers can be found in the Orange book (TCSEC). A proper balance of functionality and assurance is difficult to understand unless you look at all of the components and assess the level of risk of each as comprehensive as possible.
@dre:
Thanks for the follow-up. The thinking would be that a bare-metal hypervisor presents a much smaller attack surface than a general purpose Host OS like windows or linux and as such the chance that there would be a feasible guest–>host compromise is lower.
Also bare-metal hypervisors may not provide the
facilities that a general purpose OS which may allow an attacker to leverege such a compromise (eg, network clients, compilation tools).
The idea of vendor implemented back-doors is always a tricky one and will depend on the companies risk profile and likely threats as to whether that’s going to be a concern, but the idea of guest–>host compromise and how easy or difficult it is will, I think , have quite a large effect on the take-up of virtualization in a lot of companies..
@ Rory: Thanks for posting; great posts.
Smaller attack surfaces lower the possibility of attack, but they don’t eliminate attacks entirely. Ideally, you would be capable of responding to incidents with a vast array of tools and resources to help you for when it does happen.
Linux is just as good as any platform for this purpose. I have not seen equivalent tools such as Samhain or the99lb for other OSes. Although Windows does have some great digital forensics and e-discovery tools available, especially commercially-based.
I’ll have to disagree with your overall assessment of host-based virtualization solutions (I assume you’re speaking directly to Xen), although you do have some very attractive arguments. I like the way you think, but maybe there are other factors besides the ease in ability of compromise?
@dre.
Good point on the availability of resources, definately as far as I know specific incident/security tools for hypervisors is a pretty new field (I guess that may be one of the uses for VMwares VMSafe API). I hadn’t considered the possible positive impact of having an operating system you could leverage for security as the host…
In terms of Host based virtualization I was thinking of things like VMware server, Windows virtual server (without Hyper-V) and Xen, which in terms of approach are all roughly similar (in that they have a general purpose operating system running on the host).
@ Rory:
I’ve been giving some thought to using the host OS for security purposes for awhile. Check out the papers on virtual machine introspection (Building an IDS) at the XenAccess website. If you get a chance to read my blog post on Hardware VM security: past and present, be sure to check what I say about VMcasting as well. I think that virtualization can go a long way towards managing patches and working with (instead of against) vulnerability management and other areas of information security.
While not very comprehensive yet, Diane Barrett did a talk at LayerOne 2008 on Virtual Traces, which is digital forensics research into virtualization.