Parallel Workloads Archive: HPC2N Seth

The HPC2N Seth log

System: A Linux cluster located in Sweden
Duration: July 2002 thru January 2006
Jobs: 527,371

This log contains three and a half years worth of accounting records from the High-Performance Computing Center North (HPC2N) in Sweden. HPC2N is a joint operation with several universities and facilities. For more information about this installation, see URL http://www.hpc2n.umu.se/. This log is from a 120-node Linux cluster named Seth.

The HPC2N workload log was graciously provided by Ake Sandgren, who also helped with background information and interpretation. If you use this log in your work, please use a similar acknowledgment. Michael Jack is credited with getting it to the archive.

Downloads:

HPC2N-2002-0 13.5 MB gz original log
HPC2N-2002-2.swf 6.9 MB gz converted log
HPC2N-2002-2.2-cln.swf 3.0 MB gz cleaned log -- RECOMMENDED, see usage notes
HPC2N-2002-1.swf 6.9 MB gz old version of converted log (replaced 5 Dec 2011)
HPC2N-2002-1.1-cln.swf 3.0 MB gz old version of cleaned log (replaced 5 Dec 2011)
HPC2N-2002-2.1-cln.swf 3.0 MB gz old version of cleaned log (replaced 8 Feb 2015)
(May need to click with right mouse button to save to disk)

Papers Using this Log:

This log was used in the following papers: [feitelson08] [pascual09] [minh11] [sodan11] [vandenbossche11] [lindsay12] [kurowski12] [niu12] [krakov12] [kumar12] [klusacek12] [ababneh12] [gomezm13] [zakay13] [krakov13] [hao14] [kumar14] [zakay14] [zakay14b] [feitelson14] [lic14] [lucarelli17] [carastans17] [ntakpe17] [hai20]

System Environment

Seth is a 120-node Linux cluster.

Each node contains two 240 AMD Athlon MP2000+ processors, running at 1.667 GHz. The total system peak performance is 800 Gflops. Nodes have 1 GB of RAM each, shared by the two processors. They are connected by a 3D SCI interconnect organized as a 4x5x6 grid, and by a fast Ethernet.

Scheduling is performed with the Maui scheduler.

Log Format

The original log is available as HPC2N-2002-0.

This file contains one line per completed job and conforms to the Maui workload trace format.

Conversion Notes

The converted log is available as HPC2N-2002-2.swf. The conversion from the original format to SWF was done subject to the following. The differences between conversion 1 (reflected in HPC2N-2002-1.swf) and conversion 2 (reflected in HPC2N-2002-2.swf) are The conversion was done by a log-specific parser in conjunction with a more general converter module (version 2).

Usage Notes

The log contains abnormally high activity by several individual users. This includes unparalleled activity by user 2, who submitted 305,178 jobs, which constitutes 57.8% of the whole log, and two smaller flurries. These were removed in the cleaned version, and it is recommended that this version be used. For example, this removed the anomalous pattern of activity in different days of the week. In addition, the first 10 jobs were removed.
The cleaned log is available as HPC2N-2002-2.2-cln.swf.

A flurry is a burst of very high activity by a single user. The filters used to remove the activity of user 2 and the flurries that were identified are

user=2 (305178 jobs)
user=60 and job>254199 and job<271543 (12333 jobs)
user=67 and job>289097 and job<305173 (6984 jobs)
Removing remaining jobs up to job 10 was added in the second cleaned version, as they seem to represent activity from long before the actual logging started. In total, 324500 jobs were removed. Note that the filters were applied to the original log, and unfiltered jobs remain untouched. As a result, in the filtered log job numbering is not consecutive.

Further information on flurries and the justification for removing them can be found in:

The Log in Graphics

File HPC2N-2002-2.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance

File HPC2N-2002-2.2-cln.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance


Parallel Workloads Archive - Logs