Cluster Computing

1 Cluster Computing _____________________________________________________________________________ _

What are clusters? A cluster is a type of parallel or distributed processing system, which consists of a collection of interconnected standalone computers co - operatively working together as a single, integrated computing resource. This cluster of computers shares common network characteristics like the same namespace and it is available to other computers on the network as a single resource. These computers are linked together using high-speed network interfaces between themselves and the actual binding together of the all the individual computers in the cluster is performed by the operating system and the software used.

2 Cluster Computing _____________________________________________________________________________ _

MOTIVATION FOR CLUSTERING High cost of ‘traditional’ High Performance Computing . Clustering using Commercial Off The Shelf (COTS) is way cheaper than buying specialized machines for computing. Cluster computing has emerged as a result of the convergence of several trends, including the availability of inexpensive high performance microprocessors and high-speed networks, and the development of standard software tools for high performance distributed computing. Increased need for High Performance Computing As processing power becomes available, applications which require enormous amount of processing, like weather modeling are becoming more common place requiring the high performance computing provided by Clusters Thus the viable alternative to this problem is

3 Cluster Computing _____________________________________________________________________________ _

“Building Your Own Cluster”, which is what Cluster Computing is all about.

Components of a Cluster The main components of a cluster are the Personal Computer and the interconnection network. The computer can be built out of Commercial off the shelf components (COTS) and is available economically. The interconnection network can be either an ATM ring (Asynchronous Transfer Mode), which guarantees a fast and effective connection, or a Fast Ethernet connection, which is commonly available now. Gigabit Ethernet which provides speeds

up

to

1000Mbps,or

Myrinet

a

commercial

interconnection network with high speed and reduced latency are viable options. But for high-end scientific clustering, there are a variety of network interface cards designed specifically for clustering. Those include Myricom's Myrinet, Giganet's cLAN and the IEEE 1596 standard Scalable Coherent Interface (SCI). Those

4 Cluster Computing _____________________________________________________________________________ _

cards' function is not only to provide high bandwidth between the nodes of the cluster but also to reduce the latency (the time it takes to send messages). Those latencies are crucial to exchanging state information between the nodes to keep their operations synchronized. INTERCONNECTION NETWORKS Myricom Myricom offers cards and switches that interconnect at speeds of up to 1.28 Gbps in each direction. The cards come in two different forms, copper-based and optical. The copper version for LANs can communicate at full speed at a distance of 10 feet but can operate at half that speed at distances of up to 60 feet. Myrinet on fiber can operate at full speed up to 6.25 miles on single-mode fiber, or about 340 feet on multimode fiber. Myrinet offers only direct point-to-point, hub-based, or switchbased network configurations, but it is not limited in the number of switch fabrics that can be connected together. Adding switch fabrics simply increases the latency between nodes. The average latency between two directly connected nodes is 5 to 18 microseconds, a magnitude or more faster than Ethernet.

5 Cluster Computing _____________________________________________________________________________ _

Giganet Giganet is the first vendor of Virtual Interface (VI) architecture cards for the Linux platform, in their cLAN cards and switches. The VI architecture is a platform-neutral software and hardware system that Intel has been promoting to create clusters. It uses its own network communications protocol rather than IP to exchange data directly between the servers, and it is not intended to be a WAN routable system. The future of VI now lies in the ongoing work of the System I/O Group, which in itself is a merger of the Next-Generation I/O group led by Intel, and the Future I/O Group led by IBM and Compaq. Giganet's products

can

currently

offer

1

Gbps

unidirectional

communications between the nodes at minimum latencies of 7 microseconds.

6 Cluster Computing _____________________________________________________________________________ _

IEEE SCI The IEEE standard SCI has even lower latencies (under 2.5 microseconds), and it can run at 400 MB per second (3.2 Gbps) in each direction. SCI is a ring-topology-based networking system unlike the star topology of Ethernet. That makes it faster to communicate between the nodes on a larger scale. Even more useful is a torus topology network, with many rings between the nodes. A two-dimensional torus can be pictured as a grid of n by m nodes with a ring network at every row and every column. A three-dimensional torus is similar, with a 3D cubic grid of nodes that also has rings at every level. Supercomputing massively parallel systems use those to provide the relatively quickest path for communications between hundreds or thousands of nodes. The limiting factor in most of those systems is not the operating system or the network interfaces but the server's internal PCI bus system. Basic 32-bit, 33-MHz PCI common in nearly all desktop PCs and most low-end servers offers only 133 MB per second (1 Gbps), stunting the power of those cards. Some costly high-end servers such as the Compaq Proliant 6500 and IBM

7 Cluster Computing _____________________________________________________________________________ _

Netfinity 7000 series have 64-bit, 66-MHz cards that run at four times that speed. Unfortunately, the paradox arises that more organizations use the systems on the low end, and thus most vendors end up building and selling more of the low-end PCI cards. Specialized network cards for 64-bit, 66-MHz PCI also exist, but they come at a much higher price. For example, Intel offers a Fast Ethernet card of that sort for about $400 to $500, almost five times the price of a regular PCI version

8 Cluster Computing _____________________________________________________________________________ _

CLUSTER

CLASSIFICATION

ACCORDING

TO

ARCHITECTURE Clusters can be basically classified into two o Close Clusters o Open Clusters Close Clusters They hide most of the cluster behind the gateway node. Consequently they need less IP addresses and provide better security. They are good for computing task s. Open Clusters All nodes can be seen from outside,and hence they

need

more IPs, and cause more security concern .But they are more fl exible and are used for internet/web/information server task.

9 Cluster Computing _____________________________________________________________________________ _

Close Cluster High Speed Network

compute compute compute compute File Server node node node node node Service Network gateway gateway node

Front-end External Network

Open Cluster High Speed Network

compute compute compute compute File Server node node node node node

External Network Front-end

10 Cluster Computing _____________________________________________________________________________ _

TECHNICALITIES IN THE DESIGN OF CLUSTER Homogeneous and Heterogeneous Clusters. The cluster can either be made of homogeneous machines, machines that have the same hardware and software configurations or as a heterogeneous cluster with machines of different configuration. Heterogeneous clusters face problems of different

performance

profiles,

software

configuration

management. Diskless Versus “Disk full” Configurations This decision strongly influences what kind of networking system is used. Diskless systems are by their very nature slower performers, than machines that have local disks. This is because no matter how fast the CPU is

, the limiting factor on

performance is how fast a program can be loaded over the network. Network Selection.

11 Cluster Computing _____________________________________________________________________________ _

Speed should be the criterion for selecting the network. Channel bonding, which is a software trick that allows multiple network connections to be tied, together to increase overall performance of the system can be used to increase the performance of Ethernet networks. Security Considerations Special considerations are involved when completing the implementation of a cluster. Even with the queue system and parallel environment, extra services are required for a cluster to function as a multi-user computational platform. These services include the well-known network services NFS, NIS and rsh. NFS allows cluster nodes to share user home directories as well as installation files for the queue system and parallel environment. NIS provides correct file and process ownership across all the cluster nodes from the single source on the master machine. Although these services are significant components of a cluster, such services create numerous vulnerabilities. Thus, it would be insecure to have cluster nodes function on an open network. For these reasons, computational cluster nodes usually reside on private networks, often accessible for users only

12 Cluster Computing _____________________________________________________________________________ _

through a firewall gateway. In most cases, the firewall is configured on the master node using ipchains or iptables. Having all cluster machines on the same private network requires them to be connected to the same switch (or linked switches) and, therefore, localized at the same proximity. This situation creates a severe limitation in terms of cluster scalability. It is impossible to combine private network machines in different geographic locations into one joint cluster, because private networks are not routable with the standard Internet Protocol (IP). Combining cluster resources on different locations, so that users from various departments would be able to take advantage of available

computational

nodes,

however,

is

possible.

Theoretically, merging clusters is not only desirable but also advantageous, in the sense that different clusters are not localized at one place but are, rather, centralized. This setup provides higher availability and efficiency to clusters, and such a proposition is highly attractive. But in order to merge clusters, all the machines would have to be on a public network instead of a private one, because every single node on every cluster needs to be directly accessible from the others. If we were to do this,

13 Cluster Computing _____________________________________________________________________________ _

however, it might create insurmountable problems because of the potential--the inevitable--security breaches. We can see then that to serve scalability, we severely compromise security, but where we satisfy security concerns, scalability becomes significantly limited. Faced with such a problem, how can we make clusters scalable and, at the same time, establish a rocksolid security on the cluster networks? Enter the Virtual Private Network (VPN). VPNs often are heralded as one of the most cutting-edge, cost-saving solutions to various applications, and they are widely deployed in the areas of security, infrastructure expansion and inter-networking. A VPN adds more dimension to networking and infrastructure because it enables private networks to be connected in secure and robust ways. Private networks generally are not accessible from the Internet and are networked only within confined locations. The technology behind VPNs, however, changes what we have previously known about private networks. Through effective use of a VPN, we are able to connect previously unrelated private networks or individual hosts both securely and transparently. Being able to connect private networks opens a

14 Cluster Computing _____________________________________________________________________________ _

whole slew of new possibilities. With a VPN, we are not limited to resources in only one location (a single private network). We can finally take advantage of resources and information from all other private networks connected via VPN gateways, without having to largely change what we already have in our networks. In many cases, a VPN is an invaluable solution to integrate and better utilize fragmented resources. In our environment, the VPN plays a significant role in combining high performance Linux computational clusters located on separate private networks into one large cluster. The VPN, with its power to transparently combine two private networks through an existing open network, enabled us to connect seamlessly two unrelated clusters in different physical locations. The VPN connection creates a tunnel between gateways that allows hosts on two different subnets (e.g., 192.168.1.0/24 and 192.168.5.0/24) to see each other as if they are on the same network. Thus, we were able to operate critical network services such as NFS, NIS, rsh and the queue system over two different private networks, without compromising security over the open network. Furthermore, the VPN encrypts all the data being passed through the established tunnel and

15 Cluster Computing _____________________________________________________________________________ _

makes the network more secure and less prone to malicious exploits. The VPN solved not only the previously discussed problems with security, but it also opened a new door for scalability. Since all the cluster nodes can reside in private networks and operate through the VPN, the entire infrastructure can be better organized and the IP addresses can be efficiently managed, resulting in a more scalable and much cleaner network. Before VPNs, it was a pending problem to assign public IP addresses to every single node on the cluster, which limited the maximum number of nodes that can be added to the cluster. Now, with a VPN, our cluster can expand in greater magnitude and scale in an organized manner. As can be seen, we have successfully integrated the VPN technology to our networks and have addressed important issues of scalability, accessibility and security in cluster computing.

16 Cluster Computing _____________________________________________________________________________ _

Beowulf Cluster Basically, the Beowulf architecture is a multi-computer architecture that is used for parallel computation applications. Therefore, Beowulf clusters are primarily meant only for processor-intensive and number crunching applications and definitely not for storage applications. Primarily, a Beowulf cluster consists of a server computer that controls the functioning of many client nodes that are connected together with Ethernet or any other network comprising of a network of switches or hubs. One good feature of Beowulf is that all the system's components are available from off-the-shelf component and there is no special hardware that is required to implement it. It also uses commodity software - most often Linux - and other commonly available components like Parallel Virtual Machine (PVM) and Messaging Passing Interface (MPI). Besides serving all the client nodes in the Beowulf cluster, the server node also acts as a gateway to external users and passes files to the Beowulf system. The server is also used to drive the console of the system from where the various

17 Cluster Computing _____________________________________________________________________________ _

parameters and configuration can be monitored. In some cases, especially in very large Beowulf configurations, there is sometimes more than one server node with other specialized nodes that perform tasks like monitoring stations and additional consoles. In disk-less configurations, very often, the individual client nodes do not even know their own addresses until the server node informs them. The major difference between the Beowulf clustering system and the more commonly implemented Cluster of Workstations (CoW) is the fact that Beowulf systems tend to appear as an entire unit to the external world and not as individual workstations. In most cases, the individual workstations do not even have a keyboard, mouse or monitor and are accessed only by remote login or through a console terminal. In fact, a Beowulf node can be conceptualized as a CPU+memory package that can be plugged into the Beowulf system - much like would be done with a motherboard. It's important to realize that Beowulf is not a specific set of components or a networking topology or even a specialized kernel. Instead, it's simply a technology for clustering together Linux computers to form a parallel, virtual supercomputer.

18 Cluster Computing _____________________________________________________________________________ _

PROGRAMMING THE BEOWULF CLUSTER Parallel programming requires skill and creativity and may be more challenging than assembling the hardware of a Beowulf system. The most common model for programming Beowulf clusters is a master-slave arrangement. In this model, one node acts as the master, directing the computations performed by one or more tiers of slave nodes. Another challenge is balancing the processing workload among the cluster's PCs. Programming of a Beowulf cluster can be done in three ways o Using parallel message passing library such as PVM and MPI o Using parallel language such as High Performance Fortran and OpenMP o Using parallel math library

19 Cluster Computing _____________________________________________________________________________ _

PVM - Parallel Virtual Machines A message passing interface from Oak Ridge National Laboratory and University of Tennessee.It appeared before MPI. It is flexible for non-dedicated cluster, and is easy to use.It has l ower performance and less feature rich compared to MPI MPI - Message Passing Interface A standard message passing interface for programming cluster or parallel system from MPI Forum. It is easy to use. Since there libraries are available on many platforms and these are the defacto standards used for implementing parallel programs, programs written with PVM or MPI will run with little or no modification on large-scale machines, if the need arises.

20 Cluster Computing _____________________________________________________________________________ _

Top 10 reasons to prefer MPI over PVM MPI has more than one freely available, quality implementation. There are at least LAM, MPICH and CHIMP. The choice of development tools is not coupled to the programming interface. MPI defines a 3rd party profiling mechanism. A tool builder can extract profile information from MPI applications by supplying the MPI standard profile interface in a separate library, without ever having access to the source code of the main implementation. o MPI has full asynchronous communication. o Immediate send and receive operations can fully overlap computation. o MPI groups are solid, efficient, and deterministic. o Group membership is static. There are no race conditions caused by processes independently entering and leaving a

21 Cluster Computing _____________________________________________________________________________ _

group. New group formation is collective and group membership information is distributed, not centralized. o MPI efficiently manages message buffers. o Messages are sent and received from user data structures, not from staging buffers within the communication library. Buffering may, in some cases, be totally avoided. o MPI synchronization protects the user from 3rd party software. o All communication within a particular group of processes is marked with an extra synchronization variable, allocated by the system. Independent software products within the same process do not have to worry about allocating message tags. o MPI can efficiently program MPP and clusters. A virtual topology reflecting the communication pattern of the application can be associated with a group of processes. An MPP implementation of MPI could use that information to match processes to processors in a way that optimizes communication paths. MPI is totally portable. Recompile and run on any implementation. With virtual topologies and efficient buffer management, for example, an

22 Cluster Computing _____________________________________________________________________________ _

application moving from a cluster to an MPP could even expect good performance. MPI is formally specified. Implementations have to live up to a published document of precise semantics. MPI is a standard. Its features and behaviour were arrived at by consensus in an open forum. It can change only by the same process. Advantages and Disadvantages of Programming Using Message passing The main advantages are that these are standards, and hence portable. They provide high performance as compared to the other approaches. The disadvantage is that programming is quite difficult. Programming Using Parallel Languages There are hundreds of parallel languages but very few of them are standard. The most popular parallel language is High Performance Fortran. The High Performance Fortran Forum (HPFF), a coalition of industry, academic and laboratory representatives, works to define a set of extensions to Fortran 90

23 Cluster Computing _____________________________________________________________________________ _

known collectively as High Performance Fortran (HPF). HPF extensions provide access to high-performance architecture features while maintaining portability across platforms. The advantage if programming using parallel languages is that it is easy to code, and that it is portable. The disadvantage is lower performance and limited scalability. Programming using Parallel Math Libraries By using parallel math libraries, the complexity of writing parallel code is avoided. Some examples are PETSc, PLAPACK, ScaLAPACK math libraries. PETSc. PETSc is intended for use in large-scale application projects, and several ongoing computational science projects are built around the PETSc libraries. With strict attention to component interoperability, PETSc facilitates the integration of independently developed application modules, which often most naturally employ different coding styles and data structures. PETSc is easy to use for beginners. Moreover, its careful design allows advanced users to have detailed control over the solution process. PETSc includes an expanding suite of parallel linear and nonlinear equation solvers that are easily used in

24 Cluster Computing _____________________________________________________________________________ _

application codes written in C, C++, and Fortran. PETSc provides many of the mechanisms needed within parallel application codes, such as simple parallel matrix and vector assembly routines that allow the overlap of communication and computation. In addition, PETSc includes growing support for distributed arrays. Features include: Parallel vectors

Complete documentation

scatters

Automatic profiling of floating

gathers

point and memory usage

Parallel matrices

Consistent interface

several sparse storage formats

Intensive error checking

easy, efficient assembly.

Portable to UNIX and Windows

Scalable

parallel Over one hundred examples

preconditioners

PETSc is supported and will be

Krylov subspace methods

actively enhanced for the next

Parallel

Newton-based several years.

nonlinear solvers Parallel solvers

timestepping

(ODE)

25 Cluster Computing _____________________________________________________________________________ _

PLAPACK PLAPACK is an MPI-based Parallel Linear Algebra Package that provides an infrastructure for building parallel dense linear algebra libraries. PLAPACK provides 3 unique features. o Physically based matrix distribution o API to query matrices and vectors o Programming programming

interface

that

allows

object

oriented

26 Cluster Computing _____________________________________________________________________________ _

ScaLAPACK ScaLAPACK is a library of high performance linear algebra routines for distributed memory MIMD computers. It contains routines for solving systems of linear equations .Most machine dependencies are limited to two standard libraries called the PBLAS, or Parallel Basic Linear Algebra Subroutines, and the BLACS ,or the BLACS, or Basic Linear Algebra Communication Subroutines. LAPACK and ScaLAPACK will run on any system where the PBLAS and the BLACS are available.

27 Cluster Computing _____________________________________________________________________________ _

The traditional measure of supercomputer performance is benchmark speed: how fast the system runs a standard program. As engineers, however, we prefer to focus on how well the system can handle practical applications. Ingenuity in parallel algorithm design is more important than raw speed or capacity: in this young science, David and Goliath (or Beowulf and Grendel!) still compete on a level playing field. Beowulf systems are also muscling their way into the corporate world. Major computer vendors are now selling clusters to businesses with large computational needs. IBM, for instance, is building a cluster of 1,250 servers for NuTec Sciences, a biotechnology firm that plans to use the system to identify disease-causing genes. An equally important trend is the development of networks of PCs that contribute their processing power to a collective task. An example is SETI@home, a project

28 Cluster Computing _____________________________________________________________________________ _

launched by researchers at the University of California at Berkeley who are analyzing deep-space radio signals for signs of intelligent life. SETI@home sends chunks of data over the Internet to more than three million PCs, which process the radiosignal data in their idle time. Some experts in the computer industry predict that researchers will eventually be able to tap into a "computational grid" that will work like a power grid: users will be able to obtain processing power just as easily as they now get electricity. Above all, the Beowulf concept is an empowering force. It wrests high-level computing away from the privileged few and makes low-cost parallel-processing systems available to those with modest resources. Research groups, high schools, colleges or small businesses can build or buy their own Beowulf clusters, realizing the promise of a supercomputer in every basement.

29 Cluster Computing _____________________________________________________________________________ _

Diskless Cluster

Diskless Node

Server DHCP request Server assign IP TFTP load OS Server Supply OS Request NFS Root File System

30 Cluster Computing _____________________________________________________________________________ _

Cluster Computing: Linux Taken to the Extreme. F. M. Hoffman and W. W. Hargrove in Linux Magazine, Vol. 1, No. 1, pages 56–59; Spring 1999. Using Multivariate Clustering to Characterize Ecoregion Borders. W. W. Hargrove and F. M. Hoffman in Computers in Science and Engineering, Vol. 1, No. 4, pages 18–25; July/August 1999. How to Build a Beowulf: A Guide to the Implementation and Application of PC Clusters. Edited by T. Sterling, J. Salmon, D. J. Becker and D. F. Savarese. MIT Press, 1999. Related Links:

31 Cluster Computing _____________________________________________________________________________ _

http://stonesoup.esd.ornl.gov/ http://extremelinux.esd.ornl.gov/ http://www.beowulf.org/ http://www.cacr.caltech.edu/research/beowulf/ http://beowulf-underground.org/