Xvm Hyp Ops 2 Session 12112007

Sun(TM) xVM hypervisor Sun(TM) xVM Server 1.0 Presentation to the Sun(TM) xVM Ops Center 2 Team Todd Clayton Dec 11, 2007

1

Agenda • Introduction to Sun (TM) xVM Hypervisor • How people use hypervisor • Using hypervisor - Control domain: booting, services, tools - Guest domains: creation, booting - Debugging

• Porting Solaris • Futures – Solaris & Windows PV drivers, Sun(TM) xVM Server xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[2]

What is Sun(TM) xVM hypervisor? • • • •

An open source hypervisor A port of Solaris to run on the hypervisor A set of control tools for the hypervisor A set of support tools for running other operating systems on the hypervisor under the direction of Solaris

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[3]

Open source hypervisor technology • Originally developed at the University of Cambridge, England - Licensed under the GPLv2 and LGPL - XenSource (now Citrix): a start-up created by the original

developers of the project to commercialize the results

• Significant contributions from Intel, AMD, IBM, HP, Fujitsu, and more • Mostly x86, but also available on PPC and Itanium • Now at version 3.1.2 - Solaris port is 3.0.4-1 based, moving to 3.1.2 soon xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[4]

Hypervisor Design Principles and Goals • Existing applications and binaries must run unmodified • Support for multi-process, multi-application application environments

- Permit complex server configurations to be virtualized

within a single guest OS instance

• Paravirtualization (PV) enables high performance and strong isolation between domains - Particularly on uncooperative architectures (x86)

• Support up to 100 active VM instances on modern servers • Live migration of VM instances between servers xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[5]

Sun(TM) xVM Architecture

AGP ACPI PCI

32/64bit

dom0

domU1

VM0 Device Manager & Control s/w

VM1 Unmodified User Software

VM2 Unmodified User Software

GuestOS

GuestOS

GuestOS

(Solaris)

(XenLinux)

(Solaris)

Back-End

Back-End

Native Device Driver

Control IF

SMP

Native Device Driver

Safe HW IF

domU2

Front-End Device Drivers

Event Channel

Virtual CPU

domU3 VM3 Unmodified User Software Unmodified GuestOS (WinXP))

Front-End Device Drivers

VT or SVM

Virtual MMU

Hypervisor Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE)

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[6]

Key Capabilities • Checkpoint/restart and live migration - Managed provisioning - Grid operations: virtual platform

• Multiple OSes running simultaneously - Solaris, Linux, Windows - No longer a boot-time decision

• Special purpose kernels - JVM, drivers, filesystems, ...

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[7]

Agenda • Introduction to Sun (TM) xVM Hypervisor • How people use hypervisor • Using hypervisor - Control domain: booting, services, tools - Guest domains: creation, booting - Debugging

• Porting Solaris • Futures – Solaris & Windows PV drivers, Sun(TM) xVM Server xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[8]

How people use hypervisor • Good for:

- Develop and test: – – – –

Fast turn-around time (shutdown and reboot) User-level code Installation General kernel components

- Older Solaris, Microsoft, Linux, ... - “Network in a box” - Sharing canned system configurations

• Clone and snapshot of zvols

- Quickly produce multiple identical guest domains - Quickly return to a known stable state

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[9]

Agenda • Introduction to Sun (TM) xVM Hypervisor • How people use hypervisor • Using hypervisor - Control domain: booting, services, tools - Guest domains: creation, booting - Debugging

• Porting Solaris • Futures – Solaris & Windows PV drivers, Sun(TM) xVM Server xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[10]

Using xVM: Booting the control domain • Grub loads the hypervisor, kernel and boot archive:

title Solaris xVM kernel$ /boot/$ISADIR/xen.gz console=com1 com1=9600,8n1 module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix module$ /platform/i86pc/$ISADIR/boot_archive

• Hypervisor:

- Initializes, probes hardware, etc. - Creates dom0 environment around the kernel and boot

archive - Jumps to dom0 kernel

• Note: -

Extended Grub syntax to allow expansion of environment specific tokens (kernel$, module$, $ISADIR) Boot archive is separated into 32 bit and 64 bit

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[11]

Using xVM: dom0 services • svc:/system/xvm/store:default - File-based database used to store configuration of

known domains

• svc:/system/xvm/xend:default

- Long running daemon used by administrative tools to

communicate with the hypervisor - Performs much of the work of creating guest domains, migration, etc.

• svc:/system/xvm/console:default - Mediates access to guest domain consoles (badly)

• svc:/system/xvm/domains:default - Automatically creates and destroys guest domains at

service start/stop time (typically system boot/shutdown)

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[12]

Using xVM: dom0 tools (1) • xm

- Low-level xVM specific command to query the state of

the hypervisor, create domains, manipulate configuration, etc.

shocks# xm start x1 shocks# xm list Name ID Mem VCPUs State Time(s) Domain-0 0 984 2 r----810.3 x1 2 1023 1 r----9.1 shocks# xm console x1 ... x1 console login: root Password: Last login: Sat Sep 8 02:02:28 on console Sep 8 18:00:13 x1 login: ROOT LOGIN /dev/console Sun Microsystems Inc. SunOS 5.11 matrix-build-2007-08-21 October 2007 #

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[13]

Using xVM: dom0 tools (2) •

virsh

- hypervisor agnostic command to query the state of the

hypervisor, create domains, manipulate configuration, etc. –

Only xVM support for now, but Logical Domains, Zones and others coming

- Built on libvirt

: shocks#; virsh dominfo x1 Id: 2 Name: x1 UUID: b0bece06-8bee-085b-b657-dd642da0daa0 OS Type: linux State: blocked CPU(s): 1 CPU time: 98.7s Max memory: 1048576 kB Used memory: 1047540 kB : shocks#;

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[14]

Using xVM: dom0 tools (3) •

virt-install

-

Facilitate the installation of para-virtual and HVM guests Interactive or command line arguments Install off media (DVD), from an ISO, or over NFS Built on libvirt

Solaris PV Guest virt-install -n solarisPV --paravirt -r 1024 \ --nographics -f /export/solarisPV/root.img -s 16 \ -l /ws/matrix-gate/public/isos/72-0910/solarisdvd.iso

Solaris HVM Guest virt-install -n solarisHVM --hvm -r 1024 --vnc \ -f /export/solarisHVM/root.img -s 16 \ -c /ws/matrix-gate/public/isos/72-0910/solarisdvd.iso

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[15]

Using xVM: dom0 tools (3) cont'd •

virt-install

WinXP HVM Guest # virt-install -n winxp --hvm -r 1024 --vnc \ -f /export/winxp/root.img -s 16 -c /windows/media.iso • Set the VNC password property in xend's SMF configuration before starting a HVM domain which uses VNC # svccfg -s xvm/xend setprop \ config/vncpasswd = astring: \”somepwd\” # svcadm refresh xvm/xend; svcadm restart xvm/xend

• If remotely displaying the VNC session remotely, you must also set the vnc-listen property # svccfg -s xvm/xend setprop \ config/vnc-listen = astring: \”0.0.0.0\” # svcadm refresh xvm/xend; svcadm restart xvm/xend xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[16]

Using xVM: dom0 tools (4) •

virt-manager

(not yet integrated)

- Gnome desktop application for managing virtual

machines - Single physical system focus - Built on libvirt

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[17]

Using xVM: Guest domain creation • Create new guest domains using virt-install - Normal Solaris install for the guest domain, including jumpstart, etc. - Linux and HVM (e.g. Windows) install still something of a work in progress

• Acquire guest domain disk images and configuration from others

- Save the need for everyone to run through the

installation - Guest domains have relatively small configuration matrix - Clone and snapshot of ZFS volumes a powerful management tool xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[18]

Using xVM: Booting guest domains • A request to start a guest domain is passed to xend by tools (xm, virsh, ...) • Guest domain image is created “in core” by xend • Kernel image, boot archive, etc. are located and inserted into the domain image

- From local files, extracted from guest domain filesystem (pygrub), ...

• Backend devices necessary to support the domain are checked and, if necessary, created - lofi for file based disk images - Create virtual NICs

• Domain image and details passed to the hypervisor • Hypervisor completes domain creation, jumps to the kernel xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[19]

Using xVM: Debugging the hypervisor • printf() is your friend (or not)

• If the hypervisor panics, Solaris can usually take a dump - Includes the hypervisor image, which looks like a kernel

module in the dump

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[20]

Using xVM: Debugging dom0 and domU • Typical OpenSolaris tools work well - mdb, kmdb, dtrace

• The hypervisor console can be used to send a 'break' signal to domains - Type '^A^A^A' at the hypervisor console to start - Particularly useful for dom0

• Dom0 tools can be used to:

- Send a 'break' signal to guest domains: –

xm sysrq b <domain>

– –

xm dump-core <domain> mdb

- Dump the image of a guest domain, for use with mdb:

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[21]

Full Virtualization (HVM) • Some operating systems have not been paravirtualized - Microsoft, older Solaris, older Linux, OS/2 (!), ...

• New processor features to enable full virtualization - Intel VT and AMD-V –

Needs to be enabled by the BIOS, so having the right CPU may not be enough

- Trap to the hypervisor for “unsafe” instructions, memory

access, etc. –

Hypervisor emulates some effects, uses device emulation for others

• More features coming to provide more assist - Nested page tables, improved VT/AMD-V, ... xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[22]

HVM: I/O device emulation • A subset of QEMU (qemu-dm) is used to provide IO device emulation - VGA (Cirrus Logic) - IDE controller - NIC (AMD PCnet and RTL8139)

• Hardware emulation runs in user-space in dom0: - Trap on emulated hardware access by HVM domain - Hypervisor passes details to qemu-dm

emulates, signals hypervisor on completion - Hypervisor re-starts HVM domain - qemu-dm

• Performance is not great xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[23]

HVM Console access (1) • Need a way to display the emulated framebuffer:

• VNC

- qemu-dm exports the virtual framebuffer as a VNC

server - Reusable sessions - Standard VNC protocol, compatible with most viewers - Solaris has a bundled client –

java -jar /usr/share/gnome/vino/vnicclient.jar

• libSDL

- Simple X11 window shows virtual framebuffer - Grabs keyboard and mouse for guest - ctrl-alt breaks grab.

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[24]

HVM Console access (2) • Remote Desktop Protocol (RDP) - Windows has built-in RDP server –

Solaris does not, yet

- Best option for controlling Windows, even forwards audio - Enable via Windows Control Panel –

“System” -> “Remote”

- rdesktop is open source RDP client, targeting future

Solaris integration - Available for preview at /ws/matrixgate/public/bin/rdesktop

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[25]

Agenda • Introduction to Sun (TM) xVM Hypervisor • How people use hypervisor • Using hypervisor - Control domain: booting, services, tools - Guest domains: creation, booting - Debugging

• Porting Solaris • Futures – Solaris & Windows PV drivers, Sun(TM) xVM Server xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[26]

Porting Solaris • A new platform, i86xpv

- As much as possible shared with i86pc

• Platform support module replaces direct hardware access with hypervisor calls

- Page table manipulation, interrupt management, clock, ...

• Implement inter-domain protocols for PV console, disk and network IO:

- Frontend drivers fit in to standard frameworks (e.g. GLD)

as providers - Backend drivers provide access to dom0 resources

• Implement inter-domain protocols for access to configuration database xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[27]

Inter-domain protocols • hypervisor provided facilities are used to implement communication paths: - Shared memory - Transfer of ownership of memory - Send and receive of event notifications

• The hypervisor reference OS implementation defines a set of communications protocols using these facilities: - Console IO: simple character IO - Network IO: “point to point” ethernet segment - Disk IO: a simple block device

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[28]

Network Backend • Provide access to shared physical device • Early cut of Crossbow virtual NIC (VNIC) implementation domU2

domU1

dom0 xnb0

xnb1

IP vnic0

IP

IP

xnf0

xnf0

vnic1

bge0

hypervisor

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[29]

Disk Backend • Open devices using layered operations • Various options for storage: - Existing physical device • or partition - ZFS volume - SVM volume - Plain file (lofi)

• Embedded labels makes access from dom0 “difficult” xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[30]

When things go wrong • New bug categories: -

solaris/xvm/hypervisor solaris/xvm/kernel solaris/xvm/kernel-io solaris/xvm/utility

• Log files in /var/log/xen: - xend.log – logging and backtraces from the long

running daemon - xpvd-event.log – logs from backend device creation, removal, etc. xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[31]

Agenda • Introduction to Sun (TM) xVM Hypervisor • How people use hypervisor • Using hypervisor - Control domain: booting, services, tools - Guest domains: creation, booting - Debugging

• Porting Solaris • Futures – Solaris & Windows PV drivers, Sun(TM) xVM Server xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[32]

PV drivers for Solaris 10 • No PV version of Solaris 10 - IO performance using emulated hardware (IDE and

RTL8139) is poor

• Provide PV disk and network drivers for older Solaris releases • Bundled in a future Solaris 10 update • Performance of PV drivers in HVM domain looks similar to that of a fully PV guest domain

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[33]

Windows PV drivers • Planned for 2008

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[34]

Areas to explore for xVM Areas to explore

Efficient use of resources

Minimized xVM Server

Rapid provisioning

Improved scalability

Fault isolation Improved SLAs Increased security

Simplified mgmt Improved data & security mgmt Usability & adoption

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[35]

Customer Needs • Consolidate Windows, Solaris and Linux onto one or multiple systems • Give Windows users the power of Solaris, without the management complexity it is known for Feature

Benefits

Native support of VMDK and VHD file formats

Easy import and export of guest OS images from/to VMware and Microsoft

Make product easy to evaluate and install

Ability to quickly assess the value proposition and improve adoption

Provide excellent single node management Make management easy even for single node cases tool Make it easy to do live migration & Quickly replace a failing/failed node or alleviate a node under performance pressure replacement of failed nodes Make it easy to create clones and snapshots Easy template creation for rapid (re)deploy-ment, easy backup for possible rollback of guest OSes

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[36]

Customer Needs (cont'd) ●

Provide differentiating features by leveraging key Solaris technologies Feature

Benefits

Networking – performance

Get higher performance for virtualized guests than with other solutions

Networking – network bandwidth mgmt facilities

Give the customer the capability to customize network configuration to fit their service level needs

Predictive Self-Healing – identify which domains need to be migrated due to memory/cpu failure Storage/NAS – allow snapshots, cloning and backups of guest OSes and their data; secure data with encryption on the network between the NAS server and xVM server nodes Security – explore how can Virtual Trusted Platform Module ensure appropriate applications run on VMs Observability – visibility into control domain & guest OSes Validation Suite – extend HCTS (VHCTS)

xvm-hyp-ops2session-12112007

Ability to automatically avoid imminent hardware failures, and increase guest uptime

Make it easy to securely maintain and manage guest images across a network or even datacenters

Allow the customer to address Sarbanes-Oxley and restricted access requirements Understand what is going on with guest OSes and quickly pinpoint problems Get the assurance that guests will run and be supported

Sun Confidential: Internal Use Only

[37]

Market Opportunity • The current V12N market size is ~9% of the overall x86 market, and will grow to 40% in 2011 (IDC) • Server Virtualization is growing in pretty much every market segment • Sun is currently behind in all of these segments • Our target market for 1.0 is the x86 Server Consolidation space • VMware, Microsoft, RedHat, Novell and Citrix (XenSource) are the key competitors • VMware is the clear market leader with about 85% xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[38]

Strategic Fit • The product will bring enterprise class Solaris features to the Server V12N space • We want to make it easy for customers to migrate from their existing VMware deployments to xVM • Currently working on business plans - The focus is on using the channel for a large part of the

distribution and sales

• Virtualization is a key direction for SMI

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[39]

Product Dependencies - Technical Projects that are still in early development/ porting phase ➢ Xen 3.1.2 port ➢ blktap ➢ libvirt 0.3.3 and virt-install 0.3 ➢ FMA for xVM ➢ Security for xVM ➢ Crossbow ➢ Windows PV drivers ➢ Live CD and Image Packaging System (IPS) xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[40]

Product Dependencies Infrastructure ➢ ➢ ➢ ➢

➢ ➢ ➢

➢

Support plan for Indiana-based releases Reciprocal support agreement with MSFT Quality, performance & scalability runs Additional quality assurance for top-tier xVM host/guest interoperability matrix* V12N validation testsuite (VHCTS) Professional services for xVM Server 1.0 Documentation – blueprints, best practices & how-to guides Training

* Refer to the Appendix for details.

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[41]

Finding out more • xVM Server twiki -

http://perf.eng.sun.com/twiki/bin/view/Matrix/XVMServer

• OpenSolaris community -

[email protected] http://opensolaris.org/os/community/xen irc://irc.oftc.net/solaris-xen

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

[42]

Appendix

Top-tier Host/Guest Interop Matrix Host

Guest

Type of Guest (HVM, PV)

32-bit/64bit Guest

Solaris xVM 64-bit

S10 U4,U5

HVM + PV domU

64-bit

Yes

OpenSolaris/Nevada

PV domU

64-bit

Yes

Windows

XP- SP2

32-bit

Yes

Windows

Server 2003 – Standard SP1, SP2, R2

32-bit

Yes

Windows

Server 2003 – Enterprise SP1, SP2, R2

32-bit

Yes

>10 5-9 <5

xvm-hyp-ops2session-12112007

Sun Confidential: Internal Use Only

Program at risk Risks being managed Risks acceptable

[44]