Example LoadLeveler Batch Job Control File

Example LoadLeveler batch job control file example.ll:

Note: You MUST specify a NYBlue predefined partition of compute nodes on which your job will be run.

# @ job_type = bluegene
# @ class = normal
#
# The executable  that will run your parallel application should always be specified as per the next line.
# @ executable = /bgl/BlueLight/ppcfloor/bglsys/bin/mpirun
#
# The following statement assumes that readyblocks.pl indicated partition B032MA00 is available at the moment. You may
# specify any available partition, we are using B032MA00 only as an example.
# Run on 32 nodes using partition B032MA00.
# Specify partition using the following statement. This statement is the only way a partition should ever be 
# specified in a LoadLeveler job control file.
# The partition you designate here indicates the number of nodes, so note that we do not specify a
# desired number of nodes anywhere in the job control file.
# @ bg_partition = B032MA00
#
# initialdir (see the next active line) will be used as the working directory for this batch job. 
# @ initialdir = /home/johndoe/tests/c
#
# If for example your jobid is 3236, your output and error will be written in
# directory /home/johndoe/app1/runs, to files 3236.out and 3236.err respectively.
# @ output = $(jobid).out
# @ error = $(jobid).err
# 
# Maximum wall clock time for job will be 5 minutes.
# @ wall_clock_limit = 00:05:00
#
# Send email to johndoe@bnl.gov when job has completed.
# @ notification = complete
# @ notify_user = johndoe@bnl.gov
#
# Specify executable for your parallel application.
# @ arguments =  -exe /home/johndoe/tests/c/helloc.exe -cwd /home/johndoe/tests/c
#
# The next statement marks the end of the job step. This example is a one-job-step batch job,
# so this is equivalent to saying that the next statement marks the end of the batch job.
# @ queue

Notes for Batch Job Control File Example

Notice that you must specify:
# @ class = normal
Notice that you should specify the full path to your executable, -exe /home/johndoe/tests/c/helloc.exe in this sample.
Notes for # @ arguments above:
Specify -exe to provide full path to the executable (in this example helloc.exe) for your parallel application that contains MPI calls.
Use -cwd to specify the current working directory of the job as seen by the compute nodes.
# @ notification = complete
# @ notify_user = johndoe@bnl.gov
in the Batch script above causes email to be sent to you upon job completion. No notification if not specified. Other options include error, start, never, and always, described in IBM LoadLeveler Documentation.
On NYBlue, mpirun must be invoked in the batch job control file using the # @ executable statement, as shown in the sample above.
To submit script: llsubmit example.ll
Help for llq: llq -H
To list all LoadLeveler jobs: llq -b (Then note jobid)
To delete a LoadLeveler job: llcancel jobid where jobid was determined by invoking llq -b.
To display info as to why a jobid remains in the Hold, NotQueued, Idle or Deferred state: llq -s jobid
There is a 48 hour wall clock limit for all LoadLeveler jobs.
See the IBM LoadLeveler Manual for other LoadLeveler commands.

Back to User Guide Example

This site maintained by:
bgwebmaster@bnl.gov

One of ten national laboratories overseen and primarily funded by the Office of Science of the U.S. Department of Energy (DOE), Brookhaven National Laboratory conducts research in the physical, biomedical, and environmental sciences, as well as in energy technologies and national security. Brookhaven Lab also builds and operates major scientific facilities available to university, industry and government researchers. Brookhaven is operated and managed for DOE's Office of Science by Brookhaven Science Associates, a limited-liability company founded by Stony Brook University, the largest academic user of Laboratory facilities, and Battelle, a nonprofit, applied science and technology organization.
Privacy and Security Notice