The Essentials Series Virtual Security Concerns & Solutions

The Essentials Series

Virtual Security Concerns & Solutions sponsored by

by Greg Shields

Article 1: Understanding & Improving Hypervisor Security ..........................................................1  What Is the Hypervisor? ......................................................................................................1  Why Is Hypervisor Security Important? ..............................................................................2  The Security Implications of Hypervisors ...........................................................................3  Hypervisor-Based Attacks: An Example .............................................................................3  Preventing Hypervisor-Based Attacks .................................................................................4  Virtual Machine Eggs in Your Hypervisor Basket ..............................................................5  Article 2: Understanding & Improving Virtual Machine Security ..................................................6  The Added Issues of Virtual Machine Dormancy ...............................................................7  Agentless and Agent-Based Tools .......................................................................................9  New States Equals New Needs ............................................................................................9  Article 3: Understanding & Improving Virtual Network Security ................................................10  The Effect of Machine Migrations on Networking............................................................12  Implications of Cross-Virtual Machine Traffic .................................................................13  External Agent-Based Approaches Overcome Virtual Transience ...................................13 

i

Copyright Statement © 2008 Realtime Publishers, Inc. All rights reserved. This site contains materials that have been created, developed, or commissioned by, and published with the permission of, Realtime Publishers, Inc. (the “Materials”) and this site and any such Materials are protected by international copyright and trademark laws. THE MATERIALS ARE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice and do not represent a commitment on the part of Realtime Publishers, Inc or its web site sponsors. In no event shall Realtimepublishers.com, Inc. or its web site sponsors be held liable for technical or editorial errors or omissions contained in the Materials, including without limitation, for any direct, indirect, incidental, special, exemplary or consequential damages whatsoever resulting from the use of any information contained in the Materials. The Materials (including but not limited to the text, images, audio, and/or video) may not be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in whole or in part, except that one copy may be downloaded for your personal, noncommercial use on a single computer. In connection with such use, you may not modify or obscure any copyright or other proprietary notice. The Materials may contain trademarks, services marks and logos that are the property of third parties. You are not permitted to use these trademarks, services marks or logos without prior written consent of such third parties. Realtime Publishers and the Realtime Publishers logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. If you have any questions about these terms, or if you would like information about licensing materials from Realtime Publishers, please contact us via e-mail at [email protected].

ii

Article 1: Understanding & Improving Hypervisor Security A successful virtualization deployment delivers a host of compelling benefits to the IT organization—reduced costs from power and cooling, a right-sizing of resource demands to supply, and improved operational agility. But there’s a hidden risk that lurks in, around, and through virtualization environments that only the most observant are aware of today. That quiet killer is virtualization’s hypervisor itself.

What Is the Hypervisor? Although not all virtualization solutions leverage a hypervisor, those most commonly used for the processing of production workloads typically leverage the use of one. Virtualization platforms today that make use of a hypervisor are VMware ESX and Virtual Infrastructure, Microsoft Hyper-V, and Citrix XenSource, among others. Within these virtualization architectures, the hypervisor is a thin layer of code that resides between the physical hardware and any virtual machines that are hosted on the virtual server itself.

Figure 1: A hypervisor is used by many virtualization solutions; it is positioned between physical hardware and residing virtual machines.

Figure 1 shows a simplistic graphical representation of this layering. With virtualization solutions that make use of them, the job of the hypervisor is effectively to “proxy” requests by virtual machines to resources that are available on the physical hardware of the server.

1

Why Is Hypervisor Security Important? Organizations that leverage virtualization do so because virtualization’s hypervisors enable a level of commonality between virtual machines across all hypervisors of the same platform. When a virtual machine is created atop a virtualization solution’s hypervisor, that virtual machine is functionally similar to other virtual machines in terms of its emulated hardware composition. This commonality allows virtual machines from one host to function when relocated to another. It also allows virtual machines collocated on a single host to share the physical resources of that host while maintaining hard boundaries between individual virtual machines. Yet at the same time this introduction of a singular hypervisor across the IT environment adds a common codebase atop which resides all the virtualized computing resources of that IT environment. Should a security vulnerability in that common codebase be exploited through the use of malicious code, it would put each and every virtual machine at risk for failure. The pervasive nature of the hypervisor across all virtual hosts means that a single piece of malicious software that has the capability to compromise one hypervisor instance can easily compromise others. Figure 2 illustrates how the introduction of this single instance of replicating malware can quickly exploit each hypervisor in the IT environment. The result of this replication is the potential for failure of all the virtual machines atop each hypervisor. Replicating Malware

Virtual Machine

Virtual Machine

Virtual Machine

Virtual Machine

Virtual Machine

Virtual Machine

Virtual Machine

Virtual Machine

Hypervisor

Hypervisor

Hypervisor

Physical Hardware

Physical Hardware

Physical Hardware

Virtual Machine

Figure 2: The hypervisor’s pervasive nature within the virtualized environment makes it a single point of failure in the case of malicious compromise.

0 In short, the move to virtualization effectively puts an IT environment’s virtual machine eggs in a single hypervisor basket.

2

The Security Implications of Hypervisors Although hypervisor-based security exploits today remain mostly within the realm of academic research, their potential for massive environment failure makes them an important consideration for environments that have made the jump to virtualization. Hypervisors are by nature softwarebased code, which infers that they require occasional updating to patch security holes and protect against discovered vulnerabilities. Examples of this need have already been seen with hypervisors in production use today. For example, the security group Secunia has reported as of the time of this writing: •

19 advisories and 128 vulnerabilities for VMware ESX Server 3.x

•

7 advisories and 14 vulnerabilities for XenSource Xen 3.x

As with traditional operating systems (OSs), most hypervisor-based vulnerabilities are constrained by a need for administrative rights in order to accomplish their mission. Without the proper administrative rights, the hypervisor remains off-limits to exploitive code. Yet the level of maturity associated with rights and privileges in virtualization environments may not be to the same level as those already in place for Windows. If IT environments do not properly lock down privileges and console access to virtual environments with the same level of care seen with directory services, this situation exacerbates the potential for successful code exploitation through malicious software.

0 Essentially, IT environments must use the same level of due diligence with virtual environment security as with OS and directory services security. If security controls for the virtual environment are lax, there is a greater chance for infection.

Hypervisor-Based Attacks: An Example The situation described up to this point is perhaps best illustrated through a real-world example. In this example, let us examine a typical healthcare environment that has recently completed a substantial migration of server resources to virtualization. This company has standardized on a single virtualization platform due to the associated cost savings as well as the benefits to systems management. All hosts run the same version of the virtualization solution’s software. However, due to the timing of the virtualization project, hosts were brought online over a period of months. Not planned for in this environment was the need to monitor the versions of the hypervisor code itself. Although all hosts run the same version of the virtualization software, hosts that were installed later in the project quietly run a later build of the version. The changes associated with the newer build versions were incorporated to protect against a known network-based vulnerability.

3

As with many healthcare organizations, the example network here operates with a centralized data center but a federated operating model. Multiple offices and hospitals share the use of the network and services on that network. Due to this federated model, security controls are not cohesively applied in all areas. At some point during its operation, an instance of replicating malicious software was introduced into the environment which infected earlier builds of the virtualization software. The end result was a loss of virtual machines atop those hosts that were not properly secured.

Figure 3: Patch levels are critical even with hypervisors. On the left is an up-to-date hypervisor that survives the attack, while on the right is one that does not because it is not up to date.

Preventing Hypervisor-Based Attacks The previous example is a simplistic one that is quickly resolved by applying the right patches. But hypervisor-based attacks can come from many possible vectors. Adding to this complexity, vulnerabilities are likely to impact every virtualization solution available on the market today. With many environments leveraging the use of multiple virtualization solutions—each requiring its own vendor-specific tool for scanning—administrators are likely to see a significant added workload associated with verifying their virtual security. In order to best recognize and prevent hypervisor-based attacks, IT organizations should consider the following suggestions for improving their security posture for virtualized environments: •

Identifying hypervisor vulnerabilities with vulnerability assessments. Hypervisor-based attacks exist on-disk and on the network in much the same way that traditional malware does with physical OSs. Thus, locating potentially unwanted software means identifying their known heuristics. Using an effective vulnerability assessment solution, organizations can assess hypervisors for missing updates and properly plan change control windows to mitigate risks. Although the proper defense against a hypervisorbased attack is the installation of necessary patches, assessing their risk to the environment is equally critical.

•

Prevent network exposure. Virtualization environments enjoy a much greater level of agility in terms of network configuration. With essentially all virtualization solutions including virtual networking components inside the virtual server, it is possible to restrict traffic to those connections that have access to the hypervisor. Preventing all but the most critical of network connectivity to the hypervisor itself goes far in preventing networkbased attacks from impacting the hypervisor.

4

•

Segregate management networks. One way in which the previous points can be accomplished is through a logical separation of management networks from those used by virtual machines themselves. Virtual machines and their network traffic typically have no need for direct network access to those networks used for hypervisor management. Segregating traffic to networks exclusively used for hypervisor management limits hypervisor exposure.

•

Consider agent-based approaches. Although agentless approaches may involve fewer requirements for management, there are certain needs that can only be fulfilled through agent-based approaches. Scanning, patching, reporting, and suggesting improvement actions work best when agents leverage administrative access for digging deep into system processes. This is particularly the case when other securing mechanisms—such as those discussed in the earlier bullet points—are being used in the environment.

Virtual Machine Eggs in Your Hypervisor Basket Virtualization indeed promises huge improvements to cost and management flexibility but with the silent risk associated with its common hypervisor code across the environment. Thus, hosting all your virtual workloads atop that singular entity brings the need for added diligence in terms of security for the hypervisor itself. Environments that don’t move now to protect this pervasive interface will see the risk of large-scale virtual machine failure in the case of an infection event.

5

Article 2: Understanding & Improving Virtual Machine Security Article 1 of this series discusses some of the new and powerful hypervisor-based vecAtors for infection that arrive when an IT organization makes the move to virtualization. That article discusses how the hypervisor easily becomes a quiet risk to the unprepared organization. But the hypervisor itself is only one facet of the story. Along with its benefits, the move to virtualization brings about added risks associated with your virtual machines themselves. Physical servers traditionally tend to operate in the Powered On state for essentially their entire operational life cycle. Once built, a physical server is rarely rebooted and almost never down for an extended period of time due to the always-on needs for its services. On the contrary, while virtual machines can be considered functionally equivalent with traditional physical machines, their easy-to-create nature and single-file composition makes them much more likely to exist in states other than On and Operational. For example: •

Virtual machine templates, which are machines themselves, are rarely powered on.

•

Single-use virtual machines may linger on-disk past the point of their usefulness. Their presence on-disk may ultimately be forgotten over time.

•

Virtual machines based on templates rather than built from the ground up tend to not exist in the Build state like what is usually required in the early stages of physical server creation.

Figure 1 shows a graphical representation of these differences in states between the two machine types.

Figure 1: Virtual machines and physical machines tend to spend their time in much different states.

The most problematic of these states is the situation in which a virtual machine for one reason or another finds itself in an extended state of powered down. While powered down, a virtual machine is little more than a file on a disk. The accumulation of these files across the IT environment can grow to become a critical issue for organizations that lack the capability to inventory their state and keep them patched.

6

The Added Issues of Virtual Machine Dormancy With virtualization, most IT organizations are aware of the ease of creating new virtual machines. Virtualization’s “copy and paste” process for accomplishing this task speeds the process of bringing on new services to meet the demands of business. But at the same time, it introduces a set of risks to the computing environment. First is the problem of virtual machine spread. When the creation process grows absurdly simple, IT organizations can quickly find themselves awash in dozens or hundreds of new virtual machine instances to manage. Dealing with the management and licensing issues of a quickly growing server count can be a major hurdle for the unprepared organization. This first problem of spread is obvious but merely administrative. There is a more critical yet often overlooked security-related issue associated with virtual machine inventory growth. That issue is virtual machine dormancy. When virtual machines can be created quickly, there comes the increased likelihood that some will be created for short-term uses and then later discarded. These virtual machines when powered off exist as benign files in a data storage location but can later become a vector for compromise when powered on. The ever-changing nature of security threat prevention requires that functioning computers operate with a specific and up-to-date configuration to protect them against malicious threats in the environment. Consider the situation shown in graphical form in Figure 2. There it is easy to see how a virtual machine can be created and used, then powered off and forgotten, only to be later powered on without the proper security configurations needed to protect itself from compromise. As this example shows, protections were implemented for every operational computer in the environment. But those protections were missed on the machine that was powered off during the update. The dormant virtual machine—powered off and missing the update—when later powered on, becomes a risk for compromise. Time Line

Virtual Machine Created & Used

Virtual Machine Powered Off

Virtual Machine Powered On

Security Config Updated

Virtual Machine Compromised

Exploit Released

Figure 2: In the timeline of virtual machine dormancy, forgotten virtual machines are likely to be powered down during the period when critical security configurations are updated. This results in the virtual machine being unprepared for an exploit should it later be powered on.

7

When thinking about this issue of virtual machine dormancy, consider the answers to five questions that probe how your environment handles security and configuration updates in support of security: •

How are you patching? When your organization undergoes its regular patching process, is that process done using manual tools or through automated systems? More importantly, are those systems regularly probing all endpoints on the network to identify computers that have missed the deployment of a particular patch? Effective patching systems have the capability to regularly scan endpoints across the environment without the need for installed agents on each computer. By not relying on the need for installed clients, rogue computers that are not built to specification can be quickly identified and patched when they arrive in the network. For dormant virtual machines, regular scanning and patch updating ensures that any offline virtual machine is quickly updated before it can be exploited.

•

How are you updating security configurations? Security for computer systems is more than simply ensuring that the correct vendor patches are correctly installed. Firewall configurations, file system permissions, desktop and application lockdowns, and allowed/disallowed executables are all examples of security configurations that must be set and maintained over the life cycle of each computer in the environment. As with patches, your process for updating security configurations must have the capability to quickly identify and resolve areas of gap.

•

How are you monitoring your baseline? For both of the previous needs, there is the critical necessity of creating and maintaining a baseline across all computers. The process to monitor for that baseline and individual machine alignment with that baseline must include regular scanning irrespective of location, directory services membership, and operating system (OS) type.

•

How are you verifying that entitlements are current? One major factor in ensuring that baseline is in validating the users and entitlements on machines in the environment. For security as well as compliance purposes, IT organizations must ensure that dormant virtual machines are not hosting expired users or entitlements assigned to those users. Your management solution must have the minimal capability to peer into virtual machines as they power on to identify and remove expired accounts before they can be exploited.

•

How are you identifying when machines are powered on? The most crucial component of all these questions is in identifying when computers are brought online in the environment. When machines both virtual and physical can be identified and adjudication actions completed at the point of their power on, much of the risks associated with dormancy disappear. Active identification of machines going from Off or Extended Off states to On and Operational is a key need for environments that leverage virtualization.

8

Agentless and Agent-Based Tools In answering these questions, IT environments that make use of virtualization must incorporate tools that assess the risk of dormant virtual machines. That assessment requires two separate but linked phases. First, the location of dormant virtual machines—those that exist only as files on a disk—must be identified and inventoried prior to being powered on. By identifying dormant virtual machines while they remain benign, it is possible to move to the second phase. In the second phase, virtual machines must be protected from the point they are powered on until they are considered healthy. This protection prevents the effect of external attacks from impacting the unprotected machine until the proper protections can be put in place. In accomplishing this mission, both agentless and agent-based tools can be used. Agentless management platforms provide a mechanism to scan entire network environments irrespective of location. Agentless management platforms integrate with common management frameworks at the OS level as well as the virtual platform level to interrogate network endpoints for specific information. When otherwise unmanaged network endpoints become active, only through the use of agentless mechanisms can these endpoints be quickly identified.

 Part of that risk identification process should include the mapping of environment elements—virtual machines being only one example—to their relevance to the business. Best-in-class tools provide the ability to map network elements such as virtual machine composition to business applications. This gives the IT organization an easy way to identify potential threats and prioritize them based on risk, impact, and business priority.

Agent-based tools enable richer management capabilities of predetermined IT assets. With agents installed to virtual machines at the time of their build, the agents have the onboard ability to identify when they have been powered on. They can provide protections from within the virtual machine while incorporating the necessary configuration updates as identified by management servers.

New States Equals New Needs With the potential for entirely new states of operation associated with virtual machines comes the need for new tools. These tools ensure the proper configuration of machines even during extended periods of being powered off. Only by leveraging the right set of tools that enables monitoring for dormant machines can IT environments truly protect themselves against the accidental exposure that can occur by simply powering on a forgotten or expired virtual machine.

9

Article 3: Understanding & Improving Virtual Network Security A virtual network is not the same thing as a physical network. You’ve seen virtual networks before. These are the networks created inside most enterprise-class virtualization solutions for connecting virtual machines to each other and the outside world. They create linkable virtual switches that leverage a virtual host’s internal bus rather than its network card to connect virtual machines to each other. Virtual switches can also be attached to physical interfaces in the server in complex ways to route virtual machine traffic over multiple external networks. A graphical representation of this is seen in Figure 1.

VM

VM

VM

VM

Virtual Host

VM

VM

Virtual Host

Figure 1: Virtual switches connect virtual machines to each other and the outside world.

Yet these virtual networks are indeed not the same as their physical equivalents. Though they appear at first blush to provide much of the same functionality as seen with traditional networks, these networks are special. The level of functionality they provide is quite different than you’re used to seeing with your Cisco or Juniper devices. For example: •

Hubs vs. switches. Many virtual switches actually function as network hubs rather than network switches. This support varies by vendor but can be a critical difference in how you want to set up your virtual networking infrastructure within a virtual host. Switches by nature have the ability to segregate traffic across ports, isolating traffic to only its destination host. This process reduces the effect of improperly configured network cards while eliminating the ability to promiscuously sniff for traffic intended for other hosts. If your virtual network environment is not fully trusted, a hub-based virtual switch may not provide the level of security you require.

•

Layer 2 vs. layer 3. The virtual switch itself in most virtualization platforms today operates at the OSI model’s layer 2 rather than layer 3. Most physical infrastructures today have elevated their network configuration beyond the limitations of layer-2 switching due to the need for layer 3’s configurable virtual routing. With virtual switches operating at layer 2 instead of layer 3, it can be difficult for virtual routing to be correctly applied. Many virtual platforms leverage external mechanisms for the support of VLANs, yet these tend to be proprietary in nature.

10

•

Access control. Virtual switches are often wide open in terms of access control, with no capabilities for the assignment of per-port access control lists (ACL). This lack of effective access control means that external traffic destined for virtual machines within a virtual host can often only be controlled to the point of the virtual host itself—though again some proprietary VLAN support is often available. This lack of internal ACLs also means that virtual machine-to-virtual machine traffic within a virtual host is similarly uncontrollable. When virtual machines need to interconnect with other virtual machines via a controlled interface, often the only mechanism to support this need is through the creation of a third firewall-type appliance that controls that traffic between each. This firewall-type appliance can impact available resources needed for its processing and its positioning must be carefully controlled as the migration of virtual machines occurs over their operational life cycle.

•

Switches vs. routers. Virtual switches are simply that: switches. Thus, if routing support is necessary, an external device or virtual appliance that supports routing is necessary. As discussed earlier, the same elements associated with router appliances can used for virtual firewalling. As virtual switches, their native functionality often lacks the advanced protocol support used by routers for the routing of network traffic.

•

Layer-4 protection. Lastly, as layer-2 devices, virtual hardware today cannot protect individual virtual machines from traffic arriving over specific ports. Layer-4 ACLs allow traffic to route to a network endpoint but restrict that traffic to network ports that are appropriate for the services being hosted atop that endpoint. Without built-in layer-4 protection, it is challenging to lock down the traffic routed to virtual machines to only appropriate ports.

Care must be taken when incorporating the use of virtual machines in the environment when security configurations require advanced network configurations. Those network configurations may not be possible using virtual networking due to these inherent limitations.

11

The Effect of Machine Migrations on Networking Another facet of virtual networking that must be considered is how virtual networking is impacted by the effects of virtual machine migration. All enterprise-class virtualization platforms include the ability to hot migrate virtual machines from one host to another. This hot migration may occur for resource load-balancing purposes or as an automated action that occurs with the loss of a virtual host. In either case, that migration has the potential to impact networking in a number of different ways: •

Source and target host virtual networking mismatch. When a virtual machine is migrated from one host to another, there is the possibility that elements of its networking configuration at the virtual switch layer may not follow. Testing must be done to ensure that the virtualization platform successfully transfers the virtual switch configuration with the virtual machine during a migration event.

•

External networking configurations. When external networking configurations are targeted on a per-virtual host basis, there is the potential that migrated virtual machines will begin service after a migration at their target virtual host without the correct networking as seen by external network equipment. External networking equipment must be configured with the flexibility to quickly “find” the relocated virtual machine and reconfigure to support its new location.

•

Speed of convergence and name resolution update. A situation that is arguably more problematic in migrations that occur across subnets—such as in a disaster recovery situation—the timing of network convergence and name resolution must occur at a rate that is acceptable to connecting clients. Convergence embodies the amount of time required for the network as a whole to recognize the changes to routing associated with a change to the topology. Name resolution refers to the need for resolution mechanisms such as DNS to properly update across the organization with the virtual machine’s new location.

•

Inappropriate cross-virtual machine communication. Lastly are the concerns of virtual machines that should not have the ability to directly communicate with each other. A more detailed example of this type of communication is provided in the next section. These may be one business unit’s application that must not communicate with another’s due to line-of-business regulatory or legal considerations. Organizations that leverage virtualization and who operate under these restrictions must be aware of and compensate for the potential for inappropriate cross-virtual machine communication.

12

Implications of Cross-Virtual Machine Traffic The best way to explain the final bullet in the previous section is through the use of an example. Consider a financial institution that runs an internal payroll application in addition to its frontend finance tools. Although these may leverage the same technology for their operation, legal and compliance regulations may require the isolation of these two applications from each other. The internal finance application must never have direct—for example, non-firewalled or uncontrolled—communication with the front-facing application. When these two applications are hosted in a physical environment, it is relatively easy to ensure that their communication remains controlled at the network layer. Setting ACLs within the network infrastructure to prevent cross-communication is a trivial matter. Neither physical instance has the need or the ability to relocate its position on the network. Yet when those applications are virtualized, the situation changes significantly. Their initial positioning within the virtual infrastructure may keep them segregated from each other. However, over time, the virtual infrastructure’s load-balancing or high-availability features may later result in both virtual machines landing on the same virtual host. Assuming that external networking protections are put in place to appropriately isolate traffic as the machines migrate, once both virtual machines are collocated on the same host, they may begin crosscommunicating via the internal virtual switch. Considering the limitations discussed earlier, this situation could violate the required segregation of these two virtual machines. To resolve this situation, techniques can be implemented such as advanced TCP/IP security on the host or the use of a third-party agent-based solution. These solutions enable administrators to ensure that cross-communication does not occur even as virtual machines migrate from host to host.

External Agent-Based Approaches Overcome Virtual Transience Necessary in these instances are external tools that enable advanced networking functions similar to those discussed earlier. When native tools are not enough to provide the level of network agility required by the complex business, external tools may be necessary to manage network connectivity between virtual machines. Segregated solutions that incorporate vision outside the confines of the virtual infrastructure ensure that no matter where virtual machines go they are always under management. Agent-based solutions within the virtual host ensure that actions can be implemented based on the monitoring data gathered through the segregated tool.

13