The OSC Linux Cluster log

System Environment

We have a cached copy of the resource trace which describes the nodes. The format can still be found at Adaptive Computing (or cached) -- need to scroll down.

Log Format

Conversion Notes

• The original log does not contain data about allocated (or requested) processors. There is a field for requested nodes, which is always 0. In addition, the field that is supposed to contain the host list always lists only one node. We therefore used the fields for requested and allocated tasks to represent processors, based on the assumption that each task runs on a dedicated processor.
• The wallclock limit field seems to be unreliable, as its values were never more than about 3 minutes. This data was therefore ignored and not converted into the SWF.
• It seems that the log only contains information about completed jobs.
• The conversion loses the following data, that cannot be represented in the SWF:
  • The distinction between dispatch time and start time. The dispatch time was used in the conversion.
  • Required support such as network adapter, architecture, operating system, memory, and disk space.
  • Various data fields associated with the Maui scheduler.
• The following anomalies were identified in the conversion:
  • 34,314 jobs had an average CPU time higher than their runtime. In 27,596 cases the extra CPU time was larger than 1 minute.
  • 6,177 jobs were recorded as using 0 CPU time; this was changed to -1. All these jobs had "success" status.
  • One job did not have a start time, so the CPU time was used to estimate the start time and runtime.
  • One job had a large negative wait time, which was set to -1.

The differences between conversion 3 (reflected in OSC-Clust-2000-3.swf) and conversion 2 (OSC-Clust-2000-2.swf) is

• The number of processors was corrected from 57 to 178 (57 is the number of nodes)
• In one job, that didn't have a start time, the interval from submit to end was assigned to the runtime rather than to the wait time (the job was recorded as having completed successfully).

• Conversion 1 attempted to obtain the number of processors used as follows.
  The information of two columns from the original log was combined:
  1. column #2 - nodes requested, and
     
  2. column #38 - allocated host "list", which unfortunately always contains only one host rather than the entire list
     
  (see the original log format specification).
  • The above information was combined based on two undocumented assumptions
    1. that allocation resolution is in nodes, and
    2. that no single job can simultaneously use both quad-processor nodes and dual-processor nodes (only one node type can be used by each job)
    along with the fact that (according to the cluster resources specification) nodes 01 to 32 each have 4 processors, while nodes 33-57 each have 2 processors.
  • For example, if a job within the original log requested 5 nodes (column 2) and was allocated "node01" (column 38; quad), then the job's size in the SWF file will be 20 processors (5-nodes times 4-processors). On the other hand, if the job was allocated "node57" (dual), then its size in the SWF file would have been 10 (5-nodes times 2-processors).
  • The two assumptions are supported by the log itself, by the fact that column #2 of the original log (nodes-requested) is always equal to or smaller than 32. Recall that the cluster is composed of 32 quad-processor nodes and 25 dual-processor nodes, and so the fact no job requested more than 32 supports both assumptions. However, we could not get confirmation for these assumptions from the machine's administrators.
  Conversion 2 replaced all this mess with the tasks fields as noted above.
• In addition there seem to have been many other undocumented changes.

Usage Notes

A flurry is a burst of very high activity by a single user. The filter used to remove the flurries that were identified is

user=174 (44,617 jobs removed)

Further information on flurries and the justification for removing them can be found in:

• D. G. Feitelson and D. Tsafrir, ``Workload sanitation for performance evaluation''. In IEEE Intl. Symp. Performance Analysis of Systems and Software, pp. 221-230, Mar 2006.
• D. Tsafrir and D. G. Feitelson, ``Instability in parallel job scheduling simulation: the role of workload flurries''. In 20th Intl. Parallel and Distributed Processing Symp., Apr 2006.

The Log in Graphics

File OSC-Clust-2000-3.swf

File OSC-Clust-2000-3.1-cln.swf (cleaned)