Parallel Workloads Archive: LANL O2K

The LANL Origin 2000 Cluster (Nirvana) log

System:	Cluster of 16 Origin 2000 machines with 128 processors each (2048 total)
Duration:	Dec 1999 thru Apr 2000
Jobs:	122233

This log contains about 4 months worth of data derived from accounting records produced by the LSF software running on the 2048-node Origin 2000 cluster (Nirvana) at Los Alamos National Lab (LANL). This is the open partition of ASCI Blue Mountain. For more information about LANL's Advanced Computing Lab, see URL http://www.acl.lanl.gov/.

The original log version available here is a condensed log based on extensive data collected by LSF on Nirvana. The converted version is based directly on the LSF data, and has more fields. However, see the conversion notes for some caveats.

The workload log from the LANL Nirvana cluster was graciously provided by Fabrizio Petrini (fabrizio AT lanl.gov), who also helped with background information and interpretation. If you use this log in your work, please use a similar acknowledgment.

Downloads:

LANL-O2K-1999-0	2.5 MB gz	original abbreviated log
LANL-O2K-1999-2.swf	2.1 MB gz	converted log
LANL-O2K-1999-1.swf	2.0 MB gz	OLD VERSION of converted log (replaced 14 Nov 2011)

(May need to click with right mouse button to save to disk)

Papers Using this Log:

This log was used in the following papers: >br> [feitelson04b] [feitelson05c] [zilber05] [brevik06] [ranjan06] [talby07] [ranjan08] [iosup08] [feitelson08] [thebe09] [feitelson14]

Condensed Log Format

This file contains one line per completed job with the following white-space separated fields:

The number of processors requested (and probably also granted)
Job submission time in seconds from the epoch
Job start time in seconds from the epoch
The user name (surrounded by quote marks)
User CPU time used (total for all processors)
System CPU time used (total for all processors)
Maximal resident memory usage (KB, total for all processors)
Maximal virtual memory usage (KB, total for all processors)

Regrettably, the data in the last two fields is not reliable, as it sometimes refers to the script used to launch the job rather than to the parallel job itself.

The format of the original LSF accounting logs is described in their man page (cached).

Conversion Notes

The converted log is available as LANL-O2K-1999-2.swf. The conversion from the original format to SWF was done subject to the following.

The original log does not contain an explicit field specifying the job end time. The field specifying the time that the job-termination event was logged was used instead.
The CPU time is given in the original log as user time and system time. To derive the average CPU time per processor these were added, and the result divided by the number of nodes.
The project name was used in lieu of a group name. Group 1 in the conversion is "default".
The memory usage recorded is not the average per processor, but the maximum. However, it is not clear how reliable this value is.
The conversion loses the following data, that cannot be represented in the SWF:
- The list of nodes specified and used (albeit in the data individual nodes are not given, but only which of the 16 machines they belong to).
- The earliest start time and the deadline (albeit in the data only 4177 have start times and 3 have deadlines).
- The "resource requirement" and what it means.
- The job dependency.
- The pre-execution command.
- The working directory.
- Input, output, and error file names.
- The full batch command used to run the job.
- The job name. Related jobs have similar names, which also hint at their function, which is lost in the numeric encoding.
- Detailed resource usage data, including maximum shared text size, The integrals ober time of the shared text size, the shared memory size, the unshared data size, and the unshared stack size, the number of page reclaims and page faults, the number of times the process was swapped out, the number of block input and output operations, the number of characters read and written, the number of System V IPC messages sent and received, and the maximum resident memory usage.
- Detailed runtime data, including the number of signals received and the number of voluntary and involuntary context switches.
The following anomalies were identified in the conversion:
- 2576 jobs had an average CPU time higher than their runtime. In 1886 of these the difference was larger than 1 minute.
- 28100 jobs were recorded as using 0 CPU time; this was changed to -1. 21156 of these had "success" status. In an additional 5999 jobs CPU data was missing.
- 5316 jobs were recorded as using 0 memory; this was changed to -1. 221 of these had "success" status. In an additional 1914 jobs memory data was missing.
- 5016 jobs were recorded as using 0 processors; this was changed to -1. All these jobs had "failed" status.
- 4804 jobs had undefined start times. All these jobs had "failed" status.

The conversion was done by a log-specific parser in conjunction with a more general converter module.

The differences between conversion 2 (reflected in LANL-O2K-1999-2.swf) and conversion 1 (LANL-O2K-1999-1.swf) are

In conversion 1, parsing problems caused 244 jobs to be deleted from the log. The length of the converted log was therefore only 121,989 jobs. In conversion 2 these parsing problems were fixed.

Usage Notes

The log may have flurries, but this was not checked thoroughly yet.

The Log in Graphics

File LANL-O2K-1999-2.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance

Parallel Workloads Archive - Logs