Computing with New York Blue

IBM Blue Gene/L and Blue Gene/P Parallel Supercomputer at Brookhaven

New York Blue is an 18 rack Blue Gene/L and a 2 rack Blue Gene/P massively parallel supercomputer based on the IBM system-on-chip technology.

The Blue Gene/L machine

It consists of 18432 (18 x 1024) dual-processor Compute Nodes (Blue Gene chip) with each Compute Node having two standard 700 MHz PowerPC440 processors (a total of 36864). The two processors (cores) on a chip share a 1 GB of DDR memory.

The 18 racks are arranged in six (6) rows with three (3) racks each making up a 48x24x16 3D torus. The first rack on the first row (row 0) is designated as R00, while the third one on the sixth row (row 5) is R52.

In addition to the Compute Nodes, there are dedicated I/O nodes. Each I/O node provides the dedicated hardware that serves the operating system tasks to a group of compute nodes. The I/O node with the compute nodes that it serves make up a group that is referred to a pset. In New York Blue, the pset ratio (the ratio of I/O nodes to compute nodes) can be one of the following: 1:16, 1:32, 1:64 and 1:128.

New York Blue is subdivided into partitions (a.k.a blocks). Each partition has a specific size (number of nodes), type (Mesh or Torus), and pset ratio (one of the ratios mentioned above). Partitions can be predefined or created by users dynamically according to their job needs. A complete list of predefined partitions and naming conventions can be found in the User's Guide. Since the pset ratio varies throughout the machine, dynamic partitions can only be used with small size partitions, up to 512 nodes (this is the number of compute nodes on a midplane). Users will have to use predefined partitions for jobs requiring a larger number of nodes.

New York Blue partitions have access to about 360 TB (usable) General Parallel File System (GPFS) file systems that provide scalable, high performance I/O. The Pseries cluster that serves the GPFS file systems to the Blue Gene I/O nodes (IONs), the Front-End Node (FEN) and the Service Node (SN) is made up of 14 p575 8-way GPFS NSD servers running SUSE SLES 9. Each GPFS server connects to the functional network via 10-GigE FC links. Individual SATA disks are housed in seven SA29500 Data Direct Networks (DDN) disk systems racks.

A user account is needed to access and use New York Blue. Details on how to apply for an account are listed in Getting An Account.

New York Blue has one Front-End Node (FEN) that provides a direct user interface to the machine partitions. Users compile their application codes and submit their jobs on the FEN. The FEN is a Pseries p570, 8-way server with 64 GB of RAM running SuSE Linux Enterprise Server 9 (SLES 9). The New York Blue FEN, with the hostname fen.bluegene.bnl.gov, can be accessed remotely via the gateways (known as ssh.bluegene.bnl.gov outside BNL and as ssh.bluegene.bnl.local inside the BNL campus network). Logging details are provided in Logging in.

Jobs on New York Blue partitions are scheduled using Loadleveler and are run using mpirun. The User's Guide provides examples on how to compile application code and run interactive and batch jobs. In the future we plan to only allow batch jobs.

Blue Gene is not a typical parallel computer. It is meant for codes that scale well into hundreds or even thousands of processors. The processors in Blue Gene are slower than those on the BNL Linux clusters. So, unless your code scales well and so can benefit from using a lot of processors, you may be better off running it with fewer processors on e.g. a standard Linux cluster.

Last Modified: Monday, May 05, 2008
Please forward all questions about this site to: NYBlue Web Administrator

One of ten national laboratories overseen and primarily funded by the Office of Science of the U.S. Department of Energy (DOE), Brookhaven National Laboratory conducts research in the physical, biomedical, and environmental sciences, as well as in energy technologies and national security. Brookhaven Lab also builds and operates major scientific facilities available to university, industry and government researchers. Brookhaven is operated and managed for DOE’s Office of Science by Brookhaven Science Associates, a limited-liability company founded by Stony Brook University, the largest academic user of Laboratory facilities, and Battelle, a nonprofit, applied science and technology organization.

Privacy and Security Notice