Parallel Workloads Archive: LCG Grid

The LCG Grid log

System:	LHC (Large Hadron Collider) Computing Grid
Duration:	11 days in November – December 2005
Jobs:	188,041

This log contains 11 days of activity from multiple nodes that make up the LCG (Large Hadron Collider Computing Grid). For more information about this installation, see URL http://lcg.web.cern.ch/LCG/.

The MaxProcs noted in the log is the total number of processors in the grid, as stated in the accompanying paper. This does not necessarily reflect the nodes that were available to and used by the jobs in the log. Moreover, the log only contains jobs submitted via the grid resource brokers, and does not include locally generated loads. Therefore the calculation of system utilization is unreliable.

The LCG data was graciously provided by the e-Science Group of HEP at Imperial College London. If you use this log in your work, please use a similar acknowledgment. It was deposited in this repository by Hui Li (hui.li@computer.org).

This log was analyzed and described in Hui Li, Michael Muskulus, and Lex Wolters, "Modeling job arrivals in a data-intensive grid", In Job Scheduling Strategies for Parallel Processing, Eitan Frachtenberg and Uwe Schwiegelshohn, (ed.), Springer-Verlag, to appear.

Downloads:

LCG-2005-0	1.3 MB gz	original log
LCG-2005-1.swf	1.9 MB gz	converted log

(May need to click with right mouse button to save to disk)

Papers Using this Log:

This log was used in the following papers:

[li06] [shah10] [shah11] [carvalho12] [leal13]

System Environment

Users submit serial or parallel jobs to resource brokers. The resource brokers find suitable resources for carying out the computation, and send processes for execution on the different systems. The log is at the level of individual processes, and does not contain data about which processes may be part of the same parallel job. A side effect is that all jobs (actually processes) are recorded as having a size of 1.

Log Format

The original log is available as LCG-2005-0.

This file contains one line per completed job with the following space separated fields:

Unix timestamp of submit time
group name
user ID
compute element name
job runtime

Conversion Notes

The converted log is available as LCG-2005-1.swf. The conversion from the original format to SWF was done subject to the following.

The last 3 elements of the compute element are used as a proxy for the partition, based on the assumption that they may serve to identify a location that participates in the grid.
The wait time is not given, so assumed to be zero.
Timestamps are interpreted based on the London time zone, regardless of where the job actually ran, because the data was collected there.

The group names are encoded as follows:

number	name
1	lhcb
2	atlas
3	fusion
4	cms
5	zeus
6	dteam
7	geant4
8	dcms
9	hone
10	see
11	phicos
12	ilc
13	biomed
14	rgstest
15	ghep
16	alice
17	ncf
18	dzero
19	vlibu
20	emutd
21	esr
22	ildg
23	pvier
24	photon
25	babar
26	hungrid
27	vlefi
28	pheno

The conversion was done by a log-specific parser in conjunction with a more general converter module (version 2).

The Log in Graphics

File LCG-2005-1.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot

Parallel Workloads Archive - Logs