Logs of Real Parallel Workloads from Production Systems

The original logs come in different formats. Information about the individual format for each log is given to the degree available in the log's associated notes document. In addition to the original format, all logs are converted to the Standard Workload Format (SWF). Some of the logs also have cleaned versions; it is recommended that these be used when available.

To promote reproducibility, files are named according to the following convention. Each file name has two parts, identifying the dataset and the file version. The dataset identifier is constructed as <site>-<machine>-<year>. The version is a serial number and possible modifier. Number 0 is reserved for the original (unconverted) version. The number is increased by 1 for each new conversion (in case errors were found in a conversion and additional conversions were performed). Modified versions are indicated by a minor number. Thus a file name of CTC-SP2-1996-2.1-cln.swf is based on the dataset CTC-SP2-1996 (log of the SP2 machine at CTC starting from 1996), and represents a cleaned version of conversion 2 of this dataset. When using these logs, please specify exactly which file you are using, in order to enable others to reproduce your work.

Please send comments and additional information to feit@cs.huji.ac.il.

The Log Files:

#	Name		From	To	Mon	CPUs	U E	M e m	Standard Format: CLEANED				Standard Format: ORIGINAL				Original Log
									Jobs	Users	Util%	File	Jobs	Users	Util%	File
1	NASA iPSC		Oct 1993	Dec 1993	3	128			18,239	69	46.6	204K	42,264	69	46.7	400K	290K
2	LANL CM5		Oct 1994	Sep 1996	24	1,024	√	√	122,060	213	74.4	2.1M	201,387	213	75.2	2.8M	14M 3.6M
3	SDSC Par95		Dec 1994	Dec 1995	12	400			53,970	98	71.6	761K	76,872	98	71.7	1.1M	1.4M
4	SDSC Par96		Dec 1995	Dec 1996	12	400			32,135	60	75.6	487K	38,719	60	75.6	585K	831K
5	Early CTC SP2		Jun 1995	Apr 1996	10	430		√	use original				75,944	642	62.9	1.4M
6	CTC SP2		Jun 1996	May 1997	11	338	√		77,222	679	85.2	1.5M	79,302	679	85.2	1.5M	3.5M
7	LLNL T3D		Jun 1996	Sep 1996	4	256			use original				22,779	153	60.1	243K	569K 246K
8	KTH SP2		Sep 1996	Aug 1997	11	100	√		28,476	214	70.1	402K	28,489	214	70.4	402K	722K
9	SDSC SP2		Apr 1998	Apr 2000	24	128	√		59,715	437	83.4	1.2M	73,496	437	83.7	1.4M	3.1M
10	LANL O2K		Nov 1999	Apr 2000	5	2,048		√	use original				122,233	337	69.7	2.1M	2.4M
11	OSC Cluster		Jan 2000	Nov 2001	22	178			36,097	253	12.8	524K	80,714	254	13.8	1.1M	2.0M
12	SDSC BLUE		Apr 2000	Jan 2003	32	1,152	√		243,306	468	76.7	3.9M	250,440	468	76.8	4.0M	6.0M
13	Sandia Ross		Nov 2001	Jan 2005	37	1,524	√	√	57,882	203	49.9	1.1M	85,355	204	50.2	1.5M
14	HPC2N		Jul 2002	Jan 2006	42	240	√	√	202,871	257	60.1	2.9M	527,371	258	70.2	6.7M	13M
15	DAS2 fs0		Jan 2003	Jan 2004	12	144	√	√	use original				225,711	102	14.9	2.2M	14M
16	DAS2 fs1		Jan 2003	Dec 2003	12	64	√	√	use original				40,315	36	12.0	376K
17	DAS2 fs2		Jan 2003	Dec 2003	12	64	√	√	use original				66,429	52	19.5	641K
18	DAS2 fs3		Jan 2003	Dec 2003	12	64	√	√	use original				66,737	64	10.7	575K
19	DAS2 fs4		Feb 2003	Dec 2003	11	64	√	√	use original				33,795	40	14.5	311K
20	SDSC DataStar		Mar 2004	Apr 2005	13	1,664	√		96,069	460	67.2	1.6M	96,089	460	63.1	1.6M
21	LPC EGEE		Aug 2004	May 2005	9	140	√	√	234,889	56	24.4	2.6M	244,821	57	20.8	2.7M	14M
22	LCG		Nov 2005	Dec 2005	1	24,515			use original				188,041	216		1.9M	1.3M
23	SHARCNET		Dec 2005	Jan 2007	13	6,828		√	use original				1,195,242	412	43.6	18M	31M
24	SHARCNET Whale		Jun 2006	Jan 2007	7	3,072		√	use original				589,251	154	72.1	8.6M
25	LLNL uBGL		Nov 2006	Jun 2007	7	2,048	√		use original				112,611	62	56.1	732K	1.2M
26	LLNL Atlas		Nov 2006	Jun 2007	8	9,216	√		43,778	131	70.5	492K	60,332	132	64.1	633K	1.2M
27	LLNL Thunder		Jan 2007	Jun 2007	5	4,008	√		121,039	283	86.7	1.3M	128,662	283	87.9	1.4M	2.2M
28	ANL Intrepid		Jan 2009	Sep 2009	8	163,840	√		use original				68,936	236	59.6	893K
29	MetaCentrum		Dec 2008	Jun 2009	7	806		√	use original				103,656	147		1.5M	2.0M
30	PIK IPLEX		Apr 2009	Jul 2012	40	2,560		√	use original				742,965	225	38.0	11M	150M
31	RICC		May 2010	Sep 2010	5	8,192	√	√	use original				447,794	176	87.2	3.6M	4.0M
32	CEA CURIE		Feb 2011	Oct 2012	20	93,312	√		312,826	582	62.0	3.5M	773,138	722	29.3	6.4M	5.9M
33	Intel NetBatch pool A		Nov 2012	Dec 2012	1			√	use original				13,368,191	1,104		139M	576M
34	Intel NetBatch pool B		Oct 2012	Nov 2012	1			√	use original				13,085,800	1,615		136M	462M
35	Intel NetBatch pool C		Oct 2012	Nov 2012	1			√	use original				13,313,793	1,146		123M	586M
36	Intel NetBatch pool D		Oct 2012	Nov 2012	1			√	use original				9,054,066	862		92M	411M
37	UniLu Gaia		May 2014	Aug 2014	3	2,004	√	√	use original				51,987	84	47.9	837K	755K
38	MetaCentrum2		Jan 2013	Apr 2015	28	8,412	√	√	use original				5,731,100	880		56M	69M
39	CIEMAT Euler		Nov 2008	Dec 2017	110	1,920	√	√	use original				9,263,012	258	57.8	87M
40	KIT FH2		Jun 2016	Jan 2018	19	24,048	√	√	use original				114,355	166	60.4	1.2M
get a list of the current (cleaned) versions of all logs, useful for downloading with wget -i

	= Important notes associated with log
	= File to download
Mon	= Duration of log in months
UE	= Indicates whether log contains user runtime estimates    [You can add artificial estimates to logs that lack them by using this utility]
Mem	= Indicates whether log contains data about memory (requested or used)
Util%	= Utilization expressed as percentage

The ORIGNAL SWF files reflect the data "as is", with only minor modifications to make them self-consistent (for example, jobs that seem to start before they arrive are modified to equate the arrival and start times). Regrettably, the data often includes problematic and unrepresentative data, such as significant automated administrative activity or large-scale flurries of activity by single users. In order to ease the use of these logs in performance evaluations, we therefore also provide a CLEANED version of the logs, and recommend that this version be used (of course, you are urged to make your own decision regarding what cleaning if at all should be applied, but if you don't want to get into this, we suggest you use ours; this has the added benefit of ensuring your work is comparable with that of others). Specific information about the cleaning performed on each log is detailed in the "usage notes" section of the log's notes document. Further information about identifying and justifying the removal of problematic and unrepresentative data may be found in:

• D. G. Feitelson and D. Tsafrir, “Workload sanitation for performance evaluation”. In IEEE Intl. Symp. Performance Analysis of Systems and Software, pp. 221-230, Mar 2006.
• D. Tsafrir and D. G. Feitelson, “Instability in parallel job scheduling simulation: the role of workload flurries”. In 20th Intl. Parallel and Distributed Processing Symp., Apr 2006.

Last update	Tue Sep 24 11:12:17 IDT 2019
Visit count since Aug 20, 2004	[ see some access statistics here ]
Automatically generated by	./mk-logs.pl

Change log: