Lefthand Vmware Field Guide

LeftHand Networks VI3 field guide for SAN/iQ 8 SANs Overview Administrators implementing VMware® infrastructure 3 (VI3) on a LeftHand Networks® SAN should read this document in its entirety. Important configuration, best practices, and frequently asked questions are outlined to accelerate a successful deployment.

Contents Overview ......................................................................................................................................... 1 Initial iSCSI Setup of VI3 server .................................................................................................... 2 Licensing ..................................................................................................................................... 2 Networking for the Software initiator ......................................................................................... 2 Enabling the iSCSI software adapter .......................................................................................... 4 HBA connectivity and networking .............................................................................................. 4 Connecting and using iSCSI volumes ............................................................................................ 5 Creating the first iSCSI volume on the SAN .............................................................................. 5 Enabling VIPLB for performance ............................................................................................... 5 Discovery of the first iSCSI volume ........................................................................................... 5 Discovering additional volumes .................................................................................................. 6 Disconnecting iSCSI volumes from ESX or ESXi hosts ............................................................ 6 Troubleshooting Volume Connectivity ....................................................................................... 6 Creating a new datastore on the iSCSI volume ........................................................................... 7 Expanding a SAN/iQ volume and extending a datastore on the iSCSI volume ......................... 7 Snapshots, Remote IP Copy, and SmartClone volumes ................................................................. 7 Resignaturing .............................................................................................................................. 7 SAN/iQ snapshots of VI3 raw devices ........................................................................................ 8 SAN/iQ snapshots of VI3 VMFS datastores ............................................................................... 8 SAN/iQ Remote IP Copy Volumes and SRM ............................................................................ 8 SAN/iQ SmartClone Volumes .................................................................................................... 8 VMotion, Clustering, HA, DRS, and VCB ..................................................................................... 9

Page 1 of 13

Choosing Datastores and Volumes for virtual machines ................................................................ 9 Best Practices ................................................................................................................................ 10 FAQ............................................................................................................................................... 11

Initial iSCSI Setup of VI3 server Licensing In order to use software iSCSI functionality ESX servers must be licensed for iSCSI SAN use. Licensing information is viewable for each ESX or ESXi server within the virtual center client.

Networking for the Software initiator SAN connectivity via the iSCSI software adapter requires specific network configuration before it is enabled. To enable it correctly a VMkernel network must be created with sufficient network access to the iSCSI SAN. As a best practice use at least two Gigabit network adapters teamed together for performance and failover. To allow for iSCSI authentication ESX servers also require a Service Console that can route to the iSCSI SAN. If the an ESX servers current Service Consoles cannot route to the iSCSI SAN an additional Service Console should be added to the same network as the VMkernel used for iSCSI. ESXi servers do not require this as they do not have a Service Console. As a best practice the VMkernel network for iSCSI should be separate from the management and virtual networks used by virtual machines. If enough networks are available VMotion should use a separate network and VMkernel also. The ideal networking configuration for iSCSI depends on the number of Gigabit network connections available to a VI3 server. The most common; 2, 4, and 6 ports, are outlined here for reference. VI3 servers with only 2 Gigabit network ports are not ideal for iSCSI SANs connected by the software initiator due to lack of network bandwidth to ensure good performance. If performance is not a concern, VI3 servers can function well with only 2 gigabit ports. VI3 servers with only 2 Gigabit network ports should be configured with: o A single virtual switch comprised of both Gigabit ports teamed together containing:  A network for virtual machines  A service console network (ESX) or management network (ESXi)  A VMkernel network with VMotion enabled

Page 2 of 13

o As a best practice the VMkernel network’s failover order should be reversed from the rest of the port groups on the switch. This will make best use of bandwidth by favoring the second adapter for iSCSI and VMotion and favoring the first adapter for VM network traffic and management access. VI3 servers with 4 Gigabit network ports are capable of performing better by separating management and virtual machine traffic away from iSCSI and VMotion traffic. VI3 servers with 4 Gigabit network ports should be configured with: o Two virtual switches each comprised of two Gigabit ports teamed together. If possible one port from two separate Gigabit adapters should be used. For example if using two onboard Gigabit adapters and a dual port Ethernet card, team together port 0 from the onboard and port 0 from the Ethernet card. Then team together port 1 from the onboard and port 1 from the Ethernet card. This provides protection from some bus or card failures. 



The first virtual switch should have: 

A service console (ESX) or management network (ESXi)



A virtual machine network

The second virtual switch should have: 

A VMkernel network with VMotion enabled



For ESX a service console (required for iSCSI authentication, not required for ESXi)

VI3 servers with 6 Gigabit network ports are ideal for delivering performance with the software iSCSI initiator. The improvement over 4 ports is achieved by separating VMotion traffic and iSCSI traffic so they don’t have to share bandwidth. Both iSCSI and VMotion will perform better in this environment. VI3 servers with 6 Gigabit network ports should be configured with: o Three virtual switches each comprised of two Gigabit ports teamed together. If possible one port from separate Gigabit adapters should be used in each team to prevent some bus or card failures from affecting an entire virtual switch.

Page 3 of 13







The first virtual switch should have: 

A service console (ESX) or management network (ESXi)



A virtual machine network

The second virtual switch should have: 

A VMkernel network with VMotion disabled



For ESX a service console (required for iSCSI authentication, not required for ESXi)

The third virtual switch should have: 

A VMkernel network with VMotion enabled on a separate subnet than iSCSI

More than 6 Ports: If more than 6 network adapters are available more adapters can be added to the iSCSI virtual switch to increase available bandwidth or used for any other network services desired.

Enabling the iSCSI software adapter The VI3 iSCSI software adapter must be enabled. The iSCSI software adapter is managed from the Storage Adapters list of each ESX or ESXi server. When the iSCSI adapter is enabled, copy or write down the “iSCSI Name”. This is the iqn name that identifies this VI3 server and will be needed for authentication on the SAN. The VI3 server must be rebooted after enabling the iSCSI software adapter.

HBA connectivity and networking San connectivity via iSCSI HBAs enables offloading iSCSI processing from the VI3 server and booting ESX itself from the iSCSI SAN. HBAs do not require licensing or special networking within VI3 servers. The physical network for HBAs should be a dedicated Gigabit network to the SAN just like for software initiators. As a best practice use two HBA initiators (a dual port or two single port),

Page 4 of 13

each configured with a path to all iSCSI targets for failover. Configuring multiple HBA initiators to connect to the same target requires configuring authentication for each initiators iqn name on the SAN. Typically this is configured in as two SAN/iQ ”Servers” (one for each HBA initiator) each with permissions to the same volumes on the SAN.

Connecting and using iSCSI volumes Creating the first iSCSI volume on the SAN Before a VI3 server can mount a new iSCSI volume it must be created on the SAN and authentication configured to allow the server to access the volume. Use the LeftHand Networks Centralized Management Console (CMC) or CLI to create a new volume. Create a “Server” representing the ESX or ESXi server using the iqn name(s) copied from the VI3 server’s initiator(s), or CHAP as required. SAN volumes are simply assigned to “Servers” within the CMC or CLI.

Enabling VIPLB for performance Virtual IP load balancing, or just load balancing, is a setting on each SAN/iQ ”Server” that allows for iSCSI sessions to be distributed throughout the cluster of SAN/iQ nodes to maximize performance and bandwidth utilization. Most initiators support this feature including VI3 software initiators and HBAs so it is on by default in SAN/iQ 8 SANs. All VI3 initiators should have this feature enabled in a SAN/iQ SAN.

Discovery of the first iSCSI volume To discover volumes the software initiator of each ESX or ESXi host must have the Virtual IP address of the SAN/iQ cluster containing the volume added to its dynamic discover list. Then new targets can be discovered by simply rescanning the iSCSI software adapter on the ESX or ESXi host. The iSCSI session status of an ESX server can be viewed in the LeftHand Networks CMC by selecting the “Server” in question. Volumes that are connected show an IP address in the “Gateway Connection” field. If an IP addresses is not listed then something might be configured incorrectly on the ESX host or the “Server” object in the CMC.

Page 5 of 13

Discovering additional volumes A reboot of a VI3 server will completely refresh iSCSI connections and volumes included removing ones that are no longer available. Without rebooting additional volumes can be logged into and discovered by simply performing a rescan of the iSCSI software adapter or all adapters. HBAs can also add targets manually by configuring static targets.

Disconnecting iSCSI volumes from ESX or ESXi hosts The VI3 iSCSI software adapter does not have a session disconnect feature. To disconnect an iSCSI volume it must first be unassigned from the “Server” within the LeftHand Networks CMC. Unassigned volumes are not forcefully disconnected. Instead the server simply will not be allowed to login again. Rebooting an ESX or ESXi server will clear all unassigned volumes. Individual iSCSI sessions can be reset (resetsession) from the SAN/iQ 8 CLI to remove them from ESX or ESXi hosts forcefully without rebooting. Before forcefully resetting an iSCSI session from CLI all VMs accessing the volume through that session should be powered off or VMotioned to another server that will continue to access the volume.

Troubleshooting Volume Connectivity If new volumes are not showing up as expected try these troubleshooting steps o Ping the virtual IP address from the iSCSI initiator to ensure basic network connectivity  For the software initiator this can be done by logging into the VI3 service console and executing vmkping x.x.x.x and ping x.x.x.x. Both of those commands must be successful or networking is not correct for iSCSI. The vmkping ensures that a VMkernel network can reach the SAN and the ping ensures the service console can reach the SAN. Both must be able to network to the SAN to log into new volumes.  HBAs typically have their own ping utilities inside the BIOS of the HBA  ESXi has a network trouble shooting utility that can be used from KVM console to attempt a ping to the SAN. o Double check all iqn names or CHAP entries. For iSCSI authentication to work correctly these must be exact. Simplifying the iqn name to something shorter than the default can help with trouble shooting. o Make sure all “Servers” on the SAN have load balancing enabled. If there is a mix, some have it enabled and some don’t, then those that don’t might not connect to their volumes o Enable resignaturing. Some volumes that have been copied, snapshot, or restored from backup could look like a snapshot LUN to VI3. If resignaturing is not enabled VI3 will hide those volumes. o Verify that the iSCSI protocol is allowed in the firewall rules of ESX. Version 3.5 of ESX does not allow iSCSI traffic through the firewall by default in most installations.

Page 6 of 13

Creating a new datastore on the iSCSI volume Now that the VI3 server has an iSCSI SAN volume connected it can be formatted as a new VMFS data store or mounted as a raw device mapping (RDM) directly to virtual machines. New datastores are formatted from within the VMware virtual center client.

Expanding a SAN/iQ volume and extending a datastore on the iSCSI volume Both SAN/iQ volumes and VI3 datastores can be expanded or extended dynamically. If space is running low on a datastore you will want to first make the volume it is on larger. To increase the size of the SAN/iQ volume, simply edit it in the CMC and give it a larger size. SAN/iQ software will immediately change the LUN size and layout data across the cluster accordingly without affecting its availability. Once the SAN/iQ volume has been expanded the datastore on that volume can have a VMFS extent added to it to use the new space available. This expansion can only be done four times within the same volume since each extent is an additional primary partition created by VI3 on the SAN/iQ volume. If extending more than four times is necessary then an additional SAN/iQ volume would have to be used for the fifth extent. For more information on extents please refer to VMware documentation. http://www.vmware.com/support/pubs/vi_pubs.html

Snapshots, Remote IP Copy, and SmartClone volumes Resignaturing In order for VI3 servers to utilize SAN based snapshots resignaturing must be enabled on the VI3 server. If resignaturing is not

Page 7 of 13

enabled the VI3 server will report that the volume is blank and needs to be formatted even though it contains a valid datastore. Resignaturing is one of the advanced settings of an ESX or ESXi server that can be edited in the virtual center client. Be aware that some SANs cannot support resignaturing. If SAN storage other than LeftHand Networks SANs is also attached to the same VI3 server refer to VMware’s SAN configuration guide to verify resignaturing is an option. For more information on resignaturing please refer to VMware documentation. http://www.vmware.com/support/pubs/vi_pubs.html

SAN/iQ snapshots of VI3 raw devices SAN/iQ snapshots of VI3 raw devices function and are supported in exactly the same way they do for physical servers either booting from the SAN or accessing raw disks on the SAN. Detailed information about SAN/iQ snapshots and how they works for specific applications is available at http://www.lefthandnetworks.com/resource_library.aspx

SAN/iQ snapshots of VI3 VMFS datastores SAN/iQ snapshots are very useful in a VI3 environment. All virtual machines stored on a single volume can be snapshot at once and rolled back to anytime. Additionally SAN/iQ snapshots can be mounted to any VI3 server without interrupting access to the volume they are a point in time copy of.

SAN/iQ Remote IP Copy Volumes and SRM Remote IP Copy™ software allows SAN/iQ snapshots to be copied over WAN links to remote sites for disaster recovery or backup. VI3 environments can be protected by Remote IP Copy volumes on a scheduled basis and automated by VMware Site Recovery Manager for the simplest and most complete disaster recovery solution. LeftHand Networks provides a storage replication adapter (SRA) for VMware Site Recovery Manager (SRM) to seamlessly integrate Remote IP Copy volumes with a VI3 environment. For more information on Remote IP Copy volumes go to http://www.lefthandnetworks.com/resource_library.aspx For more information or to download the LeftHand Networks SRA for VMware Site Recovery Manager got to http://resources.lefthandnetworks.com/forms/VMware-LeftHand-SRADownload

SAN/iQ SmartClone Volumes SmartClone™ volumes are very useful in a VI3 environment. All virtual machines stored on a single volume can be cloned instantly and without replicating data. SmartClone volumes only consume changed data from the time the clone was Page 8 of 13

taken. This is the best way to deploy large quantities of cloned virtual machines or virtual desktops. SmartClone volumes can be used with any other SAN/iQ features such as snapshots or Remote IP Copy without limitation. SmartClone volumes are also very useful for performing tests on virtual machines by quickly reproducing them without taking up space on the SAN to actually copy them.

VMotion, Clustering, HA, DRS, and VCB The advanced VI3 features of VMotion, Clustering, HA, DRS, and VCB all require multiple VI3 servers to have access to volumes simultaneously. To enable this on a SAN/iQ volume multiple “Servers” (one for each VI3 server initiator) can be assigned to the same volume. To use multiple “Servers” simply create one for each VI3 host that will connect to the SAN in the same manner outlined in the Creating the first iSCSI volume on the SAN section. When multiple “Servers” are assigned to the same volume a warning will indicate that this is intended for clustered servers or clustered file systems such as ESX and ESXi. CHAP authentication can also be used for increased security or to represent many ESX or ESXi hosts as one “Server” in the LeftHand Networks CMC. Each ESX or ESXi server must be configured to use the correct CHAP credentials. New volumes on each VI3 server will have to be discovered as described in Discovery of the first iSCSI volume. VCB proxy servers should have their own “Server” configured on the SAN with read only access to the volumes that ESX or ESXi servers are accessing. VCB does not require write access and this prevents the Windows based VCB proxy server from inadvertently writing new signatures to VMFS volumes that are in use by ESX or ESXi.

Choosing Datastores and Volumes for virtual machines More than one virtual machine can and generally should be stored on each datastore or volume. Choosing where to put virtual machines should be driven by backup policies, virtual machines relationships to each other, and performance. In general virtual machines that do not have a relationship to one another should not be mixed on the same SAN/iQ volume. SAN/iQ features like snapshots, Remote IP Copy, and SmartClone volumes are very useful with virtual machines but will always affect all virtual machines on the same volume simultaneously. If virtual

Page 9 of 13

machines that have no relationship are mixed on a single volume those virtual machines will have to be snapshot, rolled back, remotely copied, and cloned together. Performance of virtual machines could also be affected if too many virtual machines are located on a single volume. The more virtual machines on a volume the more IO and SCSI reservation contention there is for that volume. Up to sixteen virtual machines on a single volume will function but might experience degraded performance, depending on the hardware configuration, if all VMs are booted at the same time. Four to eight virtual machines per volume is less likely to affect performance.

Best Practices Use at least two Gigabit network adapters teamed together for performance and failover of the iSCSI connection. Teaming network adapters provides redundancy for networking components such as adapters, cables, and switches. An added benefit to teaming is an increase in available IO bandwidth. Network Teams on SAN/iQ storage nodes are easily configured in the CMC by selecting 2 active links and enabling a “bond”. Balance ALB is the most common teaming method used on SAN/iQ storage nodes. Network Teams on VI3 servers are configured at the virtual switch level. Testing has shown that a VI3 server’s NIC team for iSCSI handles networks failures smoother if it has “rolling failover” enabled. The VMkernel network for iSCSI should be separate from the management and virtual networks used by virtual machines. If enough networks are available VMotion should use a separate network also. Separating networks by functionality, iSCSI, VMotion, Virtual Machines, provides higher reliability and performance of those functions.

Page 10 of 13

Enable load balancing for iSCSI for improved performance. The Load Balancing feature of a SAN/iQ “Server” allows iSCSI connections to be re-directed to the least busy storage node in the cluster. This keeps the load on storage nodes throughout the cluster as even as possible and improves performance of the SAN overall. This settings in enabled by default in SAN/iQ 8 management groups but had to be enabled explicitly in previous versions. Virtual machines that can be backed up and restored together can share the same volume. Since SAN/iQ snapshots, Remote IP Copy volumes, and SmartClone volumes work on a per volume basis it is best to group virtual machines on volumes based on their backup and restore relationships. For example, a test environment made up of a domain controller and a few application servers would be a good candidate to put on the same volume. Those could be snapshot, cloned, and restored as one unit.

FAQ I added the SAN/iQ cluster virtual IP address to VI3 server’s dynamic discovery but don’t see a new target? Most likely you just need to select “rescan” under the “configuration” tab and “storage adapters” or you have not configured authentication on the SAN correctly. Also, confirm all network configurations. Please refer to the Discovery of the first iSCSI volume section. I can’t see any new volumes after the first one? SAN/iQ software version 6.5 with patch 10004 or higher is necessary to mount multiple volumes on VI3 ESX servers. Contact [email protected] to receive the patch for 6.5. Upgrading to 6.6.00.4101 or higher is preferred. Is virtual IP load balancing supported for VI3? The VI3 software adapter and hardware adapters are supported by SAN/iQ Virtual IP Load Balancing. As a best practice this should be enabled on the authentication groups of all VI3 initiators. Load Balancing is enabled by default in SAN/iQ 8 management groups. I rolled back / mounted a snapshot and VI3 server says it needs to be formatted? You need to enable resignaturing on your VI3 servers to mount or roll back SAN based snapshots. Please refer to the Resignaturing section. What Initiator Should I use for a Virtual Machine boot volume/LUN?

Page 11 of 13

There are many ways to connect and present an iscsi volume to a virtual machine on an VI3 server. These include using the VI3 software adapter, a hardware adapter (HBA), and some guest operating systems have their own software iSCSI adapters. For VMFS datastores containing virtual machine’s definitions and virtual disk files the VI3 servers hardware or software adapter must be used. Which one, HBA vs software, is debateable. Each gives you full VI3 functionality and is supported equally. HBA’s advantages are in supporting boot from SAN and offloading iSCSI processing from the VI3 server. If boot from SAN is not neccessary then the software initiator is a good choice. The impact of iSCSI processing on modern processors is minimal. With either one performance is more a function of the quality of physical network, disk quantity, and disk rotation speed of the SAN being attached to. What Initiator Should I use for additional Raw Device Mappings or Virtual Disks? Aside from the boot LUN/Volume additional volumes should be used for storing application data. Specifically database and logs volumes for many applications require as a best practice separate volumes. These should be presented as either raw devices (RDM) through your chosen VI3 server initiator or connected as an iSCSI disk directly through the virtual machines guest operating system software initiator. Using RDMs or direct iSCSI allows these application volumes to be transported between physical and virtual servers seamlessly since they are formatted in the native file systems of the operating system (NTFS, EXT3, etc). In order to use the guest operating system initiator successfully ensure these guidelines are followed: o The guest operating system initiator is supported and listed in LeftHand Networks compatibility matrix o For good performance and failover the guest network the initiator is going to use is at least dual gigabit and separate from other virtual networks; VMkernel, VMotion, Service Console, Virtual Machine public networks, etc. o The guest operating system is using the vmxnet NIC driver from VMware tools. o The virtual machine will not be used in conjunction with VMware Site Recovery Manager (SRM). SRM does not work with volumes connected by guest initiators. How many Virtual Machines should be on a Volume or Datastore Please refer to the Choosing Datastores and Volumes for virtual machines section. Are jumbo frames supported? Jumbo frames (IP packets configured larger than the typical 1500 bytes, up to 9000) are supported by all SAN/iQ SANs. In order for Jumbo frames to be affective they must be enabled end to end including network adapters and switches. ESX version 3.5 allows configuration of Jumbo frames but does not support them for use with the VI3 software iSCSI initiator. HBAs are capable of utilizing Jumbo frames on any version of ESX.

Page 12 of 13

Is VCB (VMware Consolidated Backup) supported on iSCSI? VMware Consolidated Backup 1.0.3 and higher is fully supported on iSCSI SANs. Why does my 2TB or higher iSCSI volume show up as 0MB to the VI3 server? The maximum volume size supported by VI3 is 2047GB. What version of SAN/iQ software supports VMware VI3? SAN/iQ software 6.5 + patch 10004 will support all features except for VMotion with more than one virtual machine on a volume. SAN/iQ software 6.6 + patch 10005 support all features VMware enables for iSCSI. The SAN/iQ software version of storage nodes with the patch applied should be 6.6.00.4101 or higher. All subsequent releases, 6.6 SP1, 7, 7 SP1, and SAN/iQ 8 software, support VI3

Page 13 of 13