Sun Constellation Linux Cluster

Sun Constellation Linux Cluster: Ranger

System Name: Ranger Host Name: ranger.tacc.utexas.edu IP Address: 129.114.50.163 Operating System: Linux Number of Nodes: 3,936 Number of Processing Cores: 62,976 Total Memory: 123TB Peak Performance: 579.4TFlops Total Disk: 1.73PB (shared) 31.4TB (local)

System Overview The Sun Constellation Linux Cluster, named Ranger, is one of the largest computational resources in the world. Ranger was made possible by a grant awarded by the National Science Foundation in September 2006 to TACC and its partners including Sun Microsystems, Arizona State University, and Cornell University. Ranger entered formal production on February 4, 2008 and supports high-end computational science for NSF TeraGrid researchers throughout the United States, academic institutions within Texas, and components of the University of Texas System. The Ranger system is comprised of 3,936 16-way SMP compute nodes providing 15,744 AMD Opteron™ processors for a total of 62,976 compute cores, 123 TB of total memory and 1.7 PB of raw global disk space. It has a theoretical peak performance of 579 TFLOPS. All Ranger nodes are interconnected using InfiniBand technology in a full-CLOS topology providing a 1GB/sec point-topoint bandwidth. A 10 PB capacity archival system is available for long term storage and backups. Example pictures highlighting various components of the system are shown in Figures 1-3.

Figure 2. SunBlade x6420 motherboard (compute blade).

Figure 1. One of six Ranger rows: Management/IO Racks (front, black), Compute Rack (silver), and In-row Heat Exchanger (black).

Figure 3. InfiniBand Constellation Core Switch.

Architecture The Ranger compute and login nodes run a Linux OS and are managed by the Rocks 4.1 cluster toolkit. Two 3456 port Constellation switches provide dual-plane access between NEMs (Network Element Modules) of each 12-blade chassis. Several global, parallel Lustre file systems have been configured to target different storage needs. The configuration and features for the compute nodes, interconnect and I/O systems are described below, and summarized in Tables 1.1 through 1.3.

Ranger is a blade-based system. Each node is a SunBlade x6420 blade running a 2.6.18.8 Linux kernel. Each node contains four AMD Opteron Quad-Core 64-bit processors (16 cores in all) on a single board, as an SMP unit. The core frequency is 2.3 GHz and supports 4 floating-point operations per clock period with a peak performance of 9.2 GFLOPS/core or 128 GFLOPS/node. •

Each node contains 32 GB of memory. The memory subsystem has a 1.0 GHz HyperTransport system Bus, and 2 channels with 667 MHz DDR2 DIMMS. Each socket possesses an independent memory controller connected directly to an L3 cache. •

The interconnect topology is a 7-stage, full-CLOS fat tree with two large Sun InfiniBand Datacenter switches at the core of the fabric (each switch can support up to a maximum of 3,456 SDR InfiniBand ports). Each of the 328 compute chassis is connected directly to the 2 core switches. Twelve additional frames are also connected directly to the core switches and provide files system, administration and login capabilities. •

File systems: Ranger's file systems are hosted on 72 Sun x4500 disk servers, each containing 48 SATA drives, and six Sun x4600 metadata servers. From this aggregate space of 1.7PB, three global file systems are available for all users (see Table 1.5). •

Table 1.1 System Configuration and Performance Component Technology Performance/Size Peak Floating Point 579 TFLOPS (Peak) Operations Four Quad-Core AMD Opteron Nodes(blades) 3,936 Nodes / 62,976 Cores processors Memory Distributed 123 TB Aggregate Shared Disk Lustre, parallel File System 1.7 PB Raw 1 GB/sec unidirectional point-to-point Interconnect InfiniBand Switch bandwidth 2.3 micro-sec max MPI latency Table 1.2 SunBlade x6420 Compute Node Component Technology Sockets per Node/Cores per 4/4 (Barcelona) Socket Clock Speed 2.3 GHz Memory Per Node 32 GB System Bus HyperTransport, 6.4 GB bidirectional 2 GB DDR2/667, PC2-5300 ECC-registered Memory DIMMs

Table 1.2 SunBlade x6420 Compute Node PCI Express x8 Compact Flash 8 GB Table 1.3 Sun x4600 Login Nodes Component Technology 2 login nodes (login3.tacc.utexas.edu) (specific node selected when (login4.tacc.utexas.edu) accessing ranger.tacc.utexas.edu ) Sockets per Node/Cores per Socket 4/4 (Barcelona) Clock Speed 2.2 GHz Memory Per Node 32 GB

Table 1.4 AMD Barcelona Processor Technology 64-bit Clock Speed 2.3 GHz FP Results/Clock Period 4 Peak Performance/core 9.2 GFLOPS L3 Cache 2 MB on-die (shared) L2 Cache 4 x 512 KB L1 Cache 64 KB Table 1.5. Storage Systems Storage Class Size Architecture 8 Local Compact Flash GB/node Lustre, Sun x4500 disk Parallel 1.7 PB servers Ranch (Tape SAM-FS (Storage 2.8 PB Storage) Archive Manager)

Features not available to users (O/S only) 72 Sun x4500 I/O data servers, 6 Sun x4600 Metadata 10 GB/s connection through 4 GridFTP Servers

Table 1.6. Parallel File Systems Storage Class Size Quota (per User) Retention Policy $HOME ~100 TB 6 GB Backed up nightly; Not purged $WORK ~200 TB 350 GB Not backed up; Not purged $SCRATCH ~800 TB 400 TB Not backed up; Purged every 10 days

Student Name :- Ahmad Abu Obaid Student ID :- 20620136