Computing with New York Blue and the Brookhaven National Laboratory Blue Gene/Q
New York Blue/L, New York Blue/P, and the BNL Blue Gene/Q Parallel Supercomputer at Brookhaven
For NYBlue or BNL Blue Gene/Q technical or administrative assistance, please open a ticket with the Brookhaven National Laboratory Help Desk by sending email to firstname.lastname@example.org or calling 631- 344- 5522 .
There is also a password-protected Google groups forum only for users of New York Blue, all users of the machine are invited to join.
New York Blue is an 18 rack Blue Gene/L and a 2 rack Blue Gene/P massively parallel supercomputer based on the IBM system-on-chip technology.
The BNL Blue Gene/Q is a 1 rack IBM Blue Gene/Q (DD2 revision hardware and software) massively parallel supercomputer. Only BNL employees and BNL collaborators will be granted accounts on it.
The Blue Gene/L machine
It consists of 18432 (18 x 1024) dual-processor Compute Nodes (Blue Gene chip) with each Compute Node having two standard 700 MHz PowerPC440 processors (a total of 36864). The two processors (cores) on a chip share a 1 GB of DDR memory.
The 18 racks are arranged in six (6) rows with three (3) racks each making up a 48x24x16 3D torus. The first rack on the first row (row 0) is designated as R00, while the third one on the sixth row (row 5) is R52.
In addition to the Compute Nodes, there are dedicated I/O nodes. Each I/O node provides the dedicated hardware that serves the operating system tasks to a group of compute nodes. The I/O node with the compute nodes that it serves make up a group that is referred to a pset. In New York Blue, the pset ratio (the ratio of I/O nodes to compute nodes) can be one of the following: 1:16, 1:32, 1:64 and 1:128.
New York Blue is subdivided into partitions (a.k.a blocks). Each partition has a specific size (number of nodes), type (Mesh or Torus), and pset ratio (one of the ratios mentioned above). Partitions can be predefined or created by users dynamically according to their job needs. A complete list of predefined partitions and naming conventions can be found in the User's Guide. The pset ratio varies throughout the machine, having one of the four pset ratios mentioned above.
There are 512 compute nodes on a midplane.
New York Blue partitions have access to about 360 TB (usable) General Parallel File System (GPFS) file systems that provide scalable, high performance I/O. The Pseries cluster that serves the GPFS file systems to the Blue Gene I/O nodes (IONs), the Front-End Node (FEN) and the Service Node (SN) is made up of 14 p575 8-way GPFS NSD servers running SUSE SLES 9. Each GPFS server connects to the functional network via 10-GigE FC links. Individual SATA disks are housed in seven SA29500 Data Direct Networks (DDN) disk systems racks.
A user account is needed to access and use New York Blue. Details on how to apply for an account are listed in Getting An Account.
New York Blue has one Front-End Node (FEN) that provides a direct user interface to the machine partitions. Users compile their application codes and submit their jobs on the FEN. The FEN is a Pseries p570, 8-way server with 64 GB of RAM running SuSE Linux Enterprise Server 9 (SLES 9). The New York Blue FEN, with the hostname fen.bluegene.bnl.gov, can be accessed remotely via the gateways (known as ssh.bluegene.bnl.gov outside BNL and as ssh.bluegene.bnl.local inside the BNL campus network). Logging details are provided in Logging in.
Jobs on New York Blue partitions are scheduled using Loadleveler and are run using mpirun. The User's Guide provides examples on how to compile application code and run interactive and batch jobs. In the future we plan to only allow batch jobs.
Blue Gene is not a typical parallel computer. It is meant for codes that scale well into hundreds or even thousands of processors. The processors in Blue Gene are slower than those on the BNL Linux clusters. So, unless your code scales well and so can benefit from using a lot of processors, you may be better off running it with fewer processors on e.g. a standard Linux cluster.
Last Modified: Wednesday, July 25, 2012