Pxe

2. introduction this howto attempts to explain how to get pxe execute kickstart in order to install a lot of linux machines at the same time fully automatic and unattended. the main idea behind this installation infrastructure is simplicity, robustness, and performance. i assume a certain familiarity of the reader with linux in general and how to install and configure software on linux in particular.everything i am describing here is for red hat 6.2 and newer clients. pick the files for the version you are planning to install on your clients. the servers can be red hat 6.2 or newer. the mechanisms i am describing are designed to work with relatively strict security: i am using static ip addresses on all clients and no client has write access on a server. things may be simpler for you if your environment has not so tight restrictions. according to http://www.suse.de/~nashif/autoyast1/html/index.html, the same installation infrastructure can be used to install suse linux over the network. 3. pxe to install a client via pre-boot execution environment (pxe) you don't need a pxe server! pxe uses a bootp request to get an ip address and other network information and a bootloader program to the client. you can either use a bootp server for doing this or a dhcp and tftp server. in the following sections, i will describe how to set up a dhcp and tftp server. 4. dhcp install the dhcp server from isc ( http://www.isc.org/) by using red hat's rpm ftp://ftp.redhat.com /pub/redhat/redhat-7.2-en/os/i386/redhat/rpms/dhcp-2.0pl5-8.i386.rpm before you can start the daemon, you need to create an empty leases file: bash# touch /etc/dhcpd.leases. secondly, you need a configuration file /etc/dhcpd.conf which looks similar to this example: deny unknown-clients; not authoritative; option domain-name option domain-name-servers option subnet-mask

"slac.stanford.edu"; 192.168.0.9, 192.168.0.10; 255.255.255.0;

allow bootp; allow booting; option ip-forwarding option mask-supplier

false; false;

# no ip forwarding # don't respond to icmp mask req

subnet 192.168.0.0 netmask 255.255.255.0 { option routers 192.168.0.1;

} group { next-server tftpsrv; filename "pxelinux.0";

}

# name of your tftp server # name of the bootloader program

host node1 { hardware ethernet 00:02:b3:1d:16:e1; fixed-address 192.168.0.11; } host node2 { hardware ethernet 00:02:b3:1d:9f:87; fixed-address 192.168.0.12; }

because pxe does really use bootp, you need to have a tftp server running on the same machine where your dhcp server runs. in the above example, all clients are directed towards "next-server" tftpsrv. you can use a simple round-robin method to distribute the load evenly over several tftp server by putting the server name into each host section instead of having only one for all hosts in a group. group { filename "pxelinux.0";

# name of the bootloader program

host node1 { hardware ethernet 00:02:b3:1d:16:e1; fixed-address 192.168.0.11; next-server tftpsrv1; # name of first tftp server } host node2 { hardware ethernet 00:02:b3:1d:9f:87; fixed-address 192.168.0.12; next-server tftpsrv2; # name of second tftp server } } talk to your networking department about the dhcp protocol! ours had switched off forwarding the dhcp broadcasts over the routers. i needed to have an extra ip helper address configured in some routers because my dhcp server is not in the same subnet as the clients. while you are at it: ask them to disable "auto-negotiation" on all your machine ports. also ask them to enable "portfast" on all ports where your machines are connected. this disables the spanning tree protocol network gear uses. if you don't have both of these settings you can run into trouble with pxe during the early stages of your install. perhaps you need some sort of load balancing for your dhcp server or you are concerned about

the stability of your dhcp server and want some backup. in this case you might try to use two (or more) dhcp servers sharing the same configuration file over a shared file system (nfs or afs for example). the common configuration file makes maintaining more than one dhcp server simple and the clients will talk to the one that is available and least loaded. 5. automatic mac address detection we, at slac, are using static ip addresses for all our computers. when getting a whole bunch of new machines you need to get their mac addresses into the dhcpd.conf file. if you can use dynamic ip addresses, just designate an interval of ip addresses in your dhcpd.conf file to be handed out to your clients. however, i have written a script to detect mac address for new machines automatically and fully unattended. it works by exploiting the topological knowledge about which node is connected to which port on the switch. it uses snmp to query the bridging table of the switch the machines have to be connected to. it writes a new dhcpd.conf file which now additionally contains the newly detected mac addresses. the script currently works for cisco catalyst 6509 and 3750 switches and is available upon request from the author. here is the usage message of the script: script to query a switch for mac addresses. -- version 4.2, author: alf wachsmann -usage: mac_print.pl optional flags: -c(onfig) : configuration file. the default config file is expected to be in the same directory as this script. -p(xeprepare): link pxelinux config file in tftpboot directory for each ip address to do netboot. -r(reset): remove all links to pxelinux config file in tftpboot directory for each ip address to do localboot. -v(erbose): be a bit more chatty. -t(est): test the configuration file. -d(ebug): lots of debug output; especially usefull with -test option. -h(elp): print this help. see text at end of this script for a description of how it works. a configuration file for the script looks like this: # system type can be 'linux' or 'solaris'. # the scripts outputs either a dhcpd.conf addition for linux or # just mac and ip address for solaris $sys_type = 'linux';

# subnet (in fact: vlan) in which the new machines should be in (see below): $subnet = 0; # which switch to query: $switch = 'swh-farm8'; # which ports on which cards to look for. the order here # determines which ip address is assigned to which port! # (there are at most 48 ports per card.) # card => [from port, to port] %portdef = ( 3 => [1, 48], 4 => [1, 48], ); # range of ip addresses which should be used for this cluster of # machines. prefix is always "192.168.". # column one is the subnet. # the number of ip addresses in this range must match the number # of ports to observe! this consistency is checked by the script. %iprange = ( 0 => [11, 106] ); if you are using this script you want to use another script which checks for changes in the dhcpd.conf file and re-starts the dhcp server if necessary (the isc-dhcp server does not understand a hup signal). the script which detects the mac addresses cannot do this because it might run on a different host then the dhcp server (this means the dhcpd.conf file has to be in a shared files system (afs, nfs, dfs)). #!/bin/sh # author: (c) alf wachsmann , july 09, 2001. # usage: just start the script. ctrl-c out of it. conffile="/afs/slac/kickstart/dhcp/etc/dhcpd.conf" # get the md5 checksum of the dhcpd.conf file md5_res=`md5sum $conffile | awk '{print $1}'` /etc/rc.d/init.d/dhcpd start while true do md5_new=`md5sum $conffile | awk '{print $1}'` # if md5 checksum has changed, restart the dhcp server if [ $md5_res != $md5_new ]; then /etc/rc.d/init.d/dhcpd restart echo "restarted dhcp daemon" md5_res = $md5_new fi

# no 'sleep' here because this script is time critical done 6. bootloader pxe can load a program into the client's memory and start it. i found it the easiest to manage, if i first load a bootloader into the clients. the bootloader then loads its configuration file via tftp from the next-server (tftpsrv in the dhcpd.conf file example above). get h. peter anvin's pxelinux part from his syslinux package ( http://syslinux.zytor.com/). compile it and copy the file pxelinux.0 in your /tftp/ directory on your tftp server. the bootloader configuration file determines whether a client boots from its local hard disk or over the network. here are example configuration files for both cases: red hat 6.2 network boot (filename default.netboot-6.2): default linux serial 0,38400n8 label linux kernel vmlinuz-6.2 append console=console=tty0 ttys0,38400 load_ramdisk=1 \ initrd=initrd-6.2.img network \ ks=nfs:nfssrv:/data/kickstart/ks-6.2.cfg red hat 7.2 network boot (filename default.netboot-7.2): default linux serial 0,38400n8 label linux kernel vmlinuz-7.2 append ksdevice=eth1 console=tty0 console=ttys0,38400 load_ramdisk=1 \ initrd=initrd-7.2.img network \ ks=nfs:nfssrv:/data/kickstart/ks-72.cfg watch out for the line breaks in the "append" line. they are here for readability only the kernel cannot handle them!(side note: with the above console definitions appended to the linux kernel you are able to follow the kickstart installation on a serial console.) you might encounter problems with dhcp if your clients have multiple ethernet adapters (nics). the effect is, that pxe itself receives a dhcp offer but the booted kernel attempting a kickstart installation does not. a remedy for this problem is to add ip=dhcp to the list of kernel parameters in the append line in the default.netboot-xyz file. boot from local hard disk (filename default):

default linux label linux localboot 0 pxelinux.0 tries to read several configuration files. it uses the first one it finds. the filenames it looks for are determined by the ip address of the client it is running on. it converts the four decimal number parts of an ip address (they are devided by dots) into hexadecimal numbers and concatenates them. example: ip address 192.168.0.11 gets converted into c0 a8 00 0b (without the spaces). the search for files starts at c0a8000b and proceeds by removing one digit from the right (leaving c0a8000) and so forth. when all digits are removed it will try as last resort the filename default. on your tftp server, this algorithm can be used to tell each single machine how to boot: /tftp/pxelinux.cfg/ 192.168.0.11 192.168.0.12

c0a8000b -> default.netboot-7.2

# install red hat 7.2 on

c0a8000c -> default.netboot-6.2

# install red hat 6.2 on

default.netboot-6.2 default.netboot-7.2 default this is important if you install a lot of machines at the same time. you can watch the syslog file on your tftp server and whenever a client got its initial ram disk transmitted, you can remove the symlink for that machine from the pxelinux.cfg directory. this forces the client to load the default configuration which says: "boot from local disk!" when it reboots after kickstart is done. technically, it is not necessary to use a bootloader. pxe can load a linux kernel directly if told so by dhcp. i did not try this because of the reasons mentioned in the previous paragraph i find it less convenient: you would have to change the dhcpd.conf file to tell a machine not to do a network boot any more but a boot from the local hard disk. editing this file on the fly is more hassle than changing one symlink. 7. tftp pxe requires a special tftp server. read pxelinux.doc from the already mentioned syslinux package for details. in an earlier version of this document i suggested using tftp-hpa started from

xinetd. this combination of tools cannot reliably handle much more than 64 clients at a time! with more clients not all of them will get an answer from your tftp and you will see syslog messages like this: tftpd: read: connection refused. update: per h. peter anvin, the problem with tftp-hpa should be fixed with version 0.17. i haven't tried it myself, though. to overcome this problem i am now using atftp from ftp://ftp.mamalinux.com/pub/atftp/. this tftp server can run as stand-alone daemon which is the mode i am using. grab the source and compile it. it should compile just fine on red hat linux. start the daemon on your tftp server with some options: /usr/sbin/atftpd --daemon --no-multicast --group nobody --tftpd-timeout 0 -m 1000 /tftp depending on your local group definitions you may not need to change the group id the daemon is running under. the default group it uses is nogroup. the tftp server directory /tftp should look like this: /tftp/initrd-6.2.img initrd-7.2.img pxelinux.0 vmlinuz-6.2 vmlinuz-7.2 kickstart_end /tftp/pxelinux.cfg/c0a8000b -> default.netboot-7.2 c0a8000c -> default.netboot-6.2 default default.netboot-6-2 default.netboot-7.2 see section linux kernel, initial ram disk for how to get the files vmlinuz-6.2, vmlinuz-7.2, initrd-6.2.img, and initrd-7.2.img. if you want to use the script described in section a useful script, you should have the empty file kickstart_end in your tftp directory: touch /tftp/kickstart_end if you are again, as for the dhcp server, concerned about stability of your tftp server you can, again, put your /tftp/ directory in a shared file system (like nfs or afs) and have all your tftp servers access this shared space. the method for preventing endless re-installation loops of your clients describe in section a useful script will work even in this setting. --------------------------------------------------------------------------------

9. kickstart you need a kickstart configuration file to make kickstart work. the file needs to be in the place you have specified in your bootloader configuration file (see section bootloader). content of the file is as usual. see the kickstart howto for further help. if you want to use the script described in the next section, add the following at the very end of your kickstart configuration file: # the last thing happening in kickstart: # get a file via tftp which indicates we are done. echo "get kickstart_end" | /usr/bin/tftp tftpsrv make sure kickstart installs the tftp package on your clients! if your clients have write access on your server (for instance via nfs) you could mount this filesystem in a last step, remove the symlink for this client (see section tftp) and unmount the filesystem again. in this case you don't need the script described in the next section. 10. a useful script with your tftp server recording transfered files into syslog you can use the following script to automatically remove the symlinks in the /tftp/pxelinux.cfg/ directory for each machine. the script watches for a string like tftpd[...]: serving kickstart_end to 192.168.0.11:32768. this indicates that the machine with ip address 192.168.0.11 loaded the marker file as the last action before the reboot after kickstart is done. note that from version atftp-0.4 to atftp-0.5 the output format changed sightly. the following script matches the format of atftp-0.5 and later. start this script on your tftp server(s) with the command ./pxecfg_changer.pl press ctrl-c to terminate the program. you are now able to start an installation on a whole bunch of machines without worrying about missing one link with the consquence that the machine would do its installation again (and again (and again (...))). #!/usr/local/bin/perl -w # # author: alf wachsmann, , august 15, 2002 # name of script: pxecfg_changer.pl # version 2.0 # usage: # $path/pxecfg_changer.pl use strict; use vars qw($version $prg $path); my $debug

= 0;

my $verbose = 1; my $file = '/var/log/messages'; $version $prg $prg $path

= = =~ =

'2.0'; $0; s{(.*/)}{}; $1;

my $pxetftppath = "/tftp/pxelinux.cfg"; my $pxedefault = $pxetftppath."/default"; die("directory '$pxetftppath' does not exist!") unless (-d $pxetftppath); die("file '$pxedefault' does not exist!") unless (-f $pxedefault); my $hexip_prefix = uc(sprintf("%02x", 192).sprintf("%02x", 168)); my $pid = open(input, "/usr/bin/tail -n 1 -f $file |"); $sig{int} = $sig{term} = sub { system("kill $pid") }; while () { # this regexp matches output from atftp-0.6 that looks like this: # tftpd[9960]: serving kickstart_end to 192.168.0.11:32768 next unless $_ =~ /tftpd\[\d+\]: serving kickstart_end to 192\.168\.(\d+)\.(\d+):\d+$/; my $hexip = uc(sprintf("%02x", $1).sprintf("%02x", $2)); print "unlink($pxetftppath/$hexip_prefix$hexip)\n" if $debug|$verbose; unlink($pxetftppath."/".$hexip_prefix.$hexip) unless $debug; } 8. linux kernel, initial ram disk two components loaded onto a client are the linux kernel and the initial ram disk. for red hat 6.2 i got them out of the bootnet.img file. here is how you get to them: bash# bash# bash# bash#

mkdir /mnt/linux /mnt/linux1 mount /tmp/bootnet.img /mnt/linux -o loop cp /mnt/linux/initrd.img /tftp/initrd-6.2.img cp /mnt/linux/vmlinuz /tftp/vmlinuz-6.2

for red hat 7.1 and higher the files are already in a special directory in the distribution: bash# cp i386/images/pxeboot/initrd.img /tftp/initrd-7.2.img bash# cp i386/images/pxeboot/vmlinuz /tftp/vmlinuz-7.2 you can of course use the same method as for red hat 6.2 if you want to. starting with red hat 9 you have to use the images in the pxeboot directory because the bootdisk.img does not contain any network drivers!

the installer "anaconda" from red hat linux 6.2 cannot install over "eth1"! if you have a motherboard nic which cannot do pxe but your machine has a second nic which can do pxe (which was the case for my clients), you need to modify the "anaconda" source code. "anaconda" in red hat 7 and later has a new parameter ksdevice for exactly this purpose (see the default.netboot-7.2 example in section bootloader). here is the modification you need to make in file redhat6.2/i386/misc/src/anaconda/loader/loader.c: 1584,1585c1583 + /* modification by alf wachsmann */ + if (kickstartnetwork("eth1", &netdev, "dhcp", flags)) { --if (kickstartnetwork("eth0", &netdev, "dhcp", flags)) { you get the new loader program into your initial ram disk with the following commands: bash# bash# bash# bash#

cd i386/misc/src/anaconda/loader vi loader.c # make the changes from above make strip loader-network

bash# bash# bash# bash# bash# bash#

mkdir /mnt/linux /mnt/linux1 cp i386/images/bootnet.img /tmp/bootnet.img mount /tmp/bootnet.img /mnt/linux -o loop cp /mnt/linux/initrd.img /tmp/initrd.img.gz gzip -d /tmp/initrd.img.gz mount /tmp/initrd.img /mnt/linux1 -o loop

bash# cp i386/misc/src/anaconda/loader/loader-network /mnt/linux1/sbin/loader bash# umount /mnt/linux1 bash# gzip /tmp/initrd.img bash# cp /tmp/initrd.img.gz /tftp/initrd.img this new initial ram disk will now perform kickstart over "eth1" (and only over "eth1"). 11. putting it all together you now should have on one machine a dhcp server with ip addresses for all your client machines you are going to install a tftp server running the atftpd daemon with a /tftp directory populated with a bootloader program pxelinux.0, a compressed linux kernel vmlinuz-6.2, a compressed linux kernel vmlinuz-7.2, an initial ram disk initrd.img-6.2, an initial ram disk initrd.img-7.2, an empty marker file kickstart_end

a subdirectory pxelinux.cfg/ with three actual files: default.netboot-6.2 default.netboot-7.2 and default and one symlink for each client you want to install pointing to the default.netboot-6.2 or default.netboot-7.2 file a script running as "root" tail -n 1 -f /var/log/messages | ./pxecfg_changer.pl depending on your needs, you may need only the 6.2 or the 7.2 versions of the above files. you also need a kickstart configuration file your linux distribution served over the net via nfs, ftp or http 12. references here are all the tools and programs you need: anvin, h. peter, syslinux/pxelinux: a boot loader for linux, http://syslinux.zytor.com/. lefebvre, jean-pierre and lefebvre, remi, atftp, ftp://ftp.mamalinux.com/pub/atftp/. isc (internet software consortium), isc dynamic host configuration protocol (dhcp), http://www.isc.org/products/dhcp/. burgess, mark, cfengine, http://www.iu.hio.no/cfengine/. some useful literature: clerc, davic and vuilleumier, marc , linux remote-boot mini-howto: configuring remote-boot workstations with linux, dos, windows 95/98 and windows nt, http://cuiwww.unige.ch/info/pc/remote-boot/howto.html. intel corporation, preboot execution environment (pxe) specification version 2.1, ftp://download.intel.com/ial/wfm/pxespec.pdf. intel corporation, pxe software development kit instructions, ftp://download.intel.com/ial/wfm/pxesdk.pdf. intel corporation, pxe production development kit instructions, ftp://download.intel.com/ial/wfm/pxepdkread.pdf. wachsmann, alf, a general purpose high performance linux installation infrastructure http://www.slac.stanford.edu/~alfw/publications/slac-pub-9193.pdf (slac publication).