The MetaCentrum 2 log

System Environment

Jobs could run on processors from more than one cluster. While relatively rare, this did happen for 7011 jobs in the log.

Scheduling is done with TORQUE with a custom built scheduler, employing a system of general queues served by two scheduling servers. The scheduler uses common approaches such as backfilling and fairshare. Documentation is available on the MetaCentrum site. The main queues are as follows:

Queue	Priority	Time limit
q_2h	50	2h
q_4h	500	4h
q_1d	50	24h
q_2d	50	48h
q_4d	50	96h
q_1w	50	168h
q_2w	50	336h
q_2w_plus	50	720/1488h
backfill	20	24h
short	50	2h
normal	50	24h
long	50	720h
uv	30	96h
gpu	75	24h
gpu_long	55	168h

In addition, there are multiple special queues for specific users and groups and for administrative purposes. The full list is available in the MetaCentrum documentation.

The above data is valid for the second half of the log, from January 2014. Note that nearly all the queues have the same priority (50). The practical effect is that jobs are prioritized just by fairshare, and queues are basically only used to define various per-user/group limits. Thus, the system operates over one "virtual" queue which is ordered by fairshare. Before January 2014, there was a fixed queue ordering, where the highest priority was for "long" (70), followed by "short" (60), "normal" (50) and "backfill" (20). Fairshare was only used "locally", within a given queue. The changes in configuration are described in detail in the paper which introduced the log [klusacek15]. The change in configuration apparently led to a change in utilization as seen in the figures below.

Importantly, data about the specific requests made by users is included as an additional field in the original log. This is a ':'-separated list of properties, such as the number of nodes and cores requested, the architecture, and specific clusters to use or to avoid. The possible properties and the mapping of properties to clusters is available in the MetaCentrum documentation. This is considered important as it enables evaluations that take all these different constraints into account.

Log Format

• The list of nodes from which cores were allocated. Each node is indicated by a a string including the cluster name and node number, e.g. "minos25-1.zcu.cz". These are separated by commas (,) and surrounded bu curly braces ({,}). If multiple cores are allocated from the same node the node name will appear multiple times in the list.
• The queue name. The queue field of the SWF standard is not used.
• The job parameters as given to the "qsub" command. This is a ':'-separated list of properties, which often includes some of the following:
  • A number indicating the number of nodes requested
  • ppn=n meaning n processors (cores) per node
  • The architecture, typically x86_64
  • Cluster names to use or cluster names preceded by ^ to avoid
  • "excl" meaning nodes should be allocated exclusively even if not all cores are being used

Conversion Notes

• The original SWF format reserves a field for "partition", which is used to represent the cluster on which the job ran. In order to accommodate jobs that ran on more than one cluster this field was allowed to contain a comma-separated list of clusters.
• There is no status information so it is unknown if any jobs were killed.
• The conversion loses the following data, that cannot be represented in the SWF:
  • The number of clusters used by each job.
  • The number of nodes and cores per node requested by each job.
  • The precise list of processors allocated to the job, and which clusters and nodes they belong to.
  • The properties requested by the application.
• The following anomalies were identified in the conversion:
  • 8,975 jobs were recorded as requesting 0 memory; this was changed to -1.
  • 182,742 jobs were recorded as requesting 0 runtime; this was changed to -1.
  • 124,486 jobs got more runtime than they requested. in 27,834 cases the extra runtime was larger than 1 min.
  • The requested time did not match the queue limit in 2,141,108 jobs. This is OK as users are allowed to specify a time limit rather than rely on the queue limit.
  • All the jobs in the log passed a sanity check to ensure the length of the list of assigned processors was equal to the given number of assigned processors. this failed in 989 jobs because a full node was allocated exclusively ("excl" flag) even though not all the cores were required. The numbers did not match also in 290 other jobs.
  • Moreover, the number of allocated cores did not match the product of the requested nodes and ppn in 7,128 jobs.

Usage Notes

The log contains all the jobs that started in the logging period, which is all of 2013-2014. Some of these jobs are extremely long, as the maximal runtime allowed on this system is 30 days. Therefore edge effects may happen at both ends of the log, where the logged data does not represent the actual load faithfully. In particular, all the jobs executing in 2015 are actually leftovers from 2014.

The Log in Graphics