The MetaCentrum log

System Environment

Jobs could run on processors from more than one cluster. While relatively rare, this did happen for 586 jobs in the log.

Scheduling is done with PBSpro, employing a system of 11 queues as follows:

Queue	Priority	Time limit (hr)
q1	62	720
q2	70	720
q3	50	24
q4	60	2
q5	80	24
q6	65	720
q7	70	720
q8	70	4
q9	70	720
q10	99	720
q11	65	720

Importantly, data about failures and other special circumstances is provided together with the log. This is considered important for reliable evaluations, and in fact is the main point of the paper that introduced this log:

D. Klusacek and H. Rudova, ``The Importance of Complete Data Sets for Job Scheduling Simulations''. In Job Scheduling Strategies for Parallel Processing, Springer Verlag LNCS vol. 6253, pp. 132-153, 2010.

Log Format

This file contains one line per completed job with the following tab separated fields:

• Job ID
• User
• Queue
• Number of processors used
• Number of grid clusters used (originally called nodes)
• Properties required by the application (given as a list of property numbers)
• Memory used (KB)
• Arrival time (UTC timestamp)
• Start time (UTC timestamp)
• End time (UTC timestamp)
• Duration (seconds)
• Exit status
• List of assigned processors (space separated)

Conversion Notes

• The status 0 was taken to mean success, and was converted to 1. All other status values were converted to 0.
• 1118 jobs were recorded as using 0 memory; this was changed to -1.
• The conversion loses the following data, that cannot be represented in the SWF:
  • The number of clusters used by each job, as given in the used clusters field.
  • The precise list of processors allocated to the job, and which clusters they belong to.
  • The properties required by the application. Note that the meaning of the properties is unknown; they are simply listed as p1, p2, p3, etc.
• The following anomalies were identified in the conversion:
  • 1118 jobs were recorded as using 0 memory; this was changed to -1. 8 of them had "success" status.
  • All the jobs in the log passed the following two sanity checks: the duration was equal the difference between the start and end times, and the length of the list of assigned processors was equal the number of assigned processors.

• In the first conversion clusters data was not recovered. In the second conversion it was extracted using the CPU IDs specified for each job. For jobs that use more than one cluster, only the first one is noted.

Usage Notes

The log contains all the jobs that terminated in the logging period. Some of these jobs are extremely long, as the maximal runtime allowed on this system is 30 days. Thus some of the logged jobs may have started up to 30 days before the start of the logging period. As a result the initial portion of the log is extremely sparse. This effect also occurs (to a lesser degree) towards the end of the log, because extremely long jobs that run in this period are not logged because they did not terminate by the end of the logging period.

The Log in Graphics