Parallel Workloads Archive: SDSC Paragon

The San-Diego Supercomputer Center (SDSC) Paragon

System:	416-node Intel Paragon
Duration:	January 1995 thru December 1996
Jobs:	76872 in 1995, 38723 in 1996

This log contains two years worth of accounting records for the 416-node Intel Paragon located at the San Diego Supercompter Center (SDSC). For more information about SDSC, see URL http://www.sdsc.edu/. The original logs and some more information (including a log of downtime!) are also available directly from SDSC.

Due to historical reasons, the log is divided to two parts (one per year). These extensive logs contain information about the number of nodes, submit, start, and end times, CPU time used, NQS queues used and their limits, and user. There is no information about the application being run.

The workload logs from the SDSC Paragon were graciously provided by Reagan Moore (moore@sdsc.edu) and Allen Downey (downey@sdsc.edu), who also helped with background information and interpretation. If you use this log in your work, please use a similar acknowledgment.

You can also reference the following:
K. Windisch, V. Lo, R. Moore, D. Feitelson, and B. Nitzberg, ``A comparison of workload traces from two production parallel machines''. In 6th Symp. Frontiers Massively Parallel Comput., pp.319-326, Oct 1996.
This paper compares the Paragon '95 workload with the NASA Ames iPSC workload.

Downloads:

SDSC-Par-1995-0	1.4 MB gz	original log
SDSC-Par-1995-3.swf	1 MB gz	converted log
SDSC-Par-1995-3.1-cln.swf	0.8 MB gz	cleaned log -- RECOMMENDED, see usage notes
SDSC-Par-1996-0	0.8 MB gz	original log
SDSC-Par-1996-3.swf	0.6 MB gz	converted log
SDSC-Par-1996-3.1-cln.swf	0.5 MB gz	cleaned log -- RECOMMENDED, see usage notes
SDSC-Par-1995-1.swf	0.4 MB gz	OLD VERSION of converted log (replaced 1 Aug 2006)
SDSC-Par-1995-1.1-cln.swf	0.4 MB gz	OLD VERSION of cleaned log (replaced 1 Aug 2006)
SDSC-Par-1996-1.swf	0.4 MB gz	OLD VERSION of converted log (replaced 1 Aug 2006)
SDSC-Par-1996-1.1-cln.swf	0.4 MB gz	OLD VERSION of cleaned log (replaced 1 Aug 2006)
SDSC-Par-1995-2.swf	1 MB gz	OLD VERSION of converted log (replaced 29 Nov 2011)
SDSC-Par-1995-2.1-cln.swf	0.8 MB gz	OLD VERSION of cleaned log (replaced 29 Nov 2011)
SDSC-Par-1996-2.swf	0.6 MB gz	OLD VERSION of converted log (replaced 29 Nov 2011)
SDSC-Par-1996-2.1-cln.swf	0.5 MB gz	OLD VERSION of cleaned log (replaced 29 Nov 2011)
SDSC-Par-1995-down	7 KB gz	downtime log
SDSC-Par-1996-down	5 KB gz	downtime log

(May need to click with right mouse button to save to disk)

Papers Using these Logs:

These two logs (or partial early versions of them) were used in the following papers: [feitelson96b] [downey97a] [downey97c] [downey98b] [feitelson98b] [smith98] [downey99] [talby99b] [mualem01] [feitelson01] [lawson02] [ernemann03] [lublin03] [song04] [feitelson04b] [feitelson05c] [zilber05] [brevik06] [feitelson06a] [tsafrir06a] [franke06] [ranjan06] [talby07] [feitelson07a] [shmueli07] [liy07] [ranjan08] [iosup08] [feitelson08] [shmueli09] [feitelson09] [minh09] [thebe09] [sodan11] [yuan11] [krakov12] [kumar12] [ababneh12] [zakay14] [feitelson14]

System Environment

The Paragon is a mesh with processing nodes based on the Intel i860 processor. The SDSC system has 416 nodes, divided thus:

48	Interactive partition
352	Compute partition (for parallel jobs via NQS)
6	Service partition (logins etc.)
10	I/O partition (parallel file system)

The compute partition, in turn, is divided into 64 nodes reserved for short jobs, and 288 for long jobs.

Batch jobs were handled by NQS, which was configured with a large number of queues. Queue names are parsed as follows:

  q [f] nodes len

The optional `f' indicates the use of fat nodes, with 32MB of memory. 256 nodes in the compute partition are thus configured. Other nodes have 16MB of memory.
`Nodes' is the limit on the number of nodes used. The queues are configured with powers of two.
`Len' is the limit on the wallclock time, and is one of the following:

s	Short: 1 hour
m	Medium: 4 hours
l	Long: 12 hours

There are also two low-priority standby queues.

The scheduling algorithm used on this system is described in detail in the following paper:
Michael Wan, Reagan Moore, George Kremenek, and Ken Steube, "A Batch Scheduler for the Intel Paragon with a Non-Contiguous Node Allocation Algorithm". In Job Scheduling Strategies for Parallel Processing, Dror G. Feitelson, and Larry Rudolph (Eds.), Springer-Verlag, pp. 48-64, 1996, Lect. Notes Comput. Sci. vol. 1162.

Log Format

The original log files are available as SDSC-Par-1995-0 and SDSC-Par-1996-0. These files contains one line per completed job with the following white-space separated fields:

User ID
Machine user ID
User account
Size = number of nodes
Type = 1 if parallel batch job, 0 if interactive parallel job.
CPU-hours = number of node-hours the job executed. This is the actual job execution time and is smaller than the wall-clock time * number_of_nodes
rate = charging factor for CPU time
idle CPU-hrs = number of node-hours the entir partition is idle while reserved for a batch job. Note that a job consists of a script submitted to NQS. It is possible for a job to do file I/O in the service nodes before it starts executing. This means the compute partition can be idle until the parallel job is actually loaded into the partition.
rate = charging factor for idle time
under-used = currently not calculated
rate = not used
total charge = sum of rate times CPU time across the three categories
job submitted = time submitted to batch queue, in date hr:min:sec
job started = time job started execution, in date hr:min:sec
job ended = time job completed, in date hr:min:sec. Note wall_clock_time is equal to the job_ended time - job_started time.
status = termination status, 1 means completed successfully
reqst = ignore
CPUhrs = time limit of batch queue
queue name = queue to which job was submitted, UNKNOWN means an interactive job.

The user ID field is sanitized to preserve privacy. However, the two logs were sanitized independently. As a result, it is not clear whether user IDs are consistent from 1995 to 1996. This prevents the concatanation of the logs into a single two-year log.

Conversion Notes

The converted logs are available as SDSC-Par-1995-3.swf and SDSC-Par-1996-3.swf. The conversion from the original format to the standard workload format was done subject to the following.

Interactive jobs do not have a submit time. Their start time was therefore used as the submit time, leading to noting them as having waited for 0 time (which is fine, because they are indeed not queued).
In the original log CPU time is given in total hours. This was converted to average per processor in seconds, as mandated in the SWF.
Jobs that are identified as interactive type are postulated to have been submitted to the interactive partition (partition 1), and those identified as batch type are assumed to have been submitted to the batch partition (partition 2).
The conversion loses the following data, that cannot be represented in the SWF:
- User account and charging information.
The following anomalies were identified in the conversion:
- In 1995 the wait time was missing in 822 jobs, and in 1996 in 806 jobs.
- In 1995 11 jobs had a negative wait time (recorded as having started before being submitted), and in 1996 this happened to 17 jobs. In all but one case the difference was larger than 1 min; in fact, they ranged from about half an hour to nearly 2 days! Negative wait times of up to 1 hr 5 min were changed to 0 effectively adjusting the start and end times. Larger negative wait times were changed to -1.
- 14 jobs in 1995 and 9 in 1996 had undefined start times. In 10 and 8 cases respectively this was approximated based on the CPU time.
- In 27 jobs in 1995 and 10 in 1996 the runtime was missing and approximated using CPU time.
- In 1995 there were also 15 jobs that had large negative run times (reocorded as having ended before being started). These were all larger than 1 hr 5 min, and were changed to -1.
- 13 jobs in 1995 and 1 in 1996 had undefined end times.
- 4,960 jobs in 1995 and 1,266 in 1996 were recorded as using 0 CPU time; this was changed to -1. 4,931 of them in 1995 and 1,250 in 1996 had "success" status.
- 24,382 jobs in 1995 and 5,422 in 1996 had an average CPU time higher than their runtime. In 3061 cases in 1995 and 12 in 1996 the extra CPU time was larger than 1 minute.
- CPU time was missing in 4,960 jobs in 1995 and 1,266 jobs in 1996.

The conversion was done by a log-specific parser in conjunction with a more general converter module.

The differences between conversion 3 (reflected in SDSC-Par-1995-3.swf and SDSC-Par-1996-3.swf) and conversion 2 (SDSC-Par-1995-2.swf and SDSC-Par-1996-2.swf) are

Conversion 3 includes better handling of missing submit and start times. In particular, availalbe data regarding CPU time is used to approximate them. This affected a few dozen jobs in each year.

The differences between conversion 2 (reflected in SDSC-Par-1995-2.swf and SDSC-Par-1996-2.swf) and conversion 1 (SDSC-Par-1995-1.swf and SDSC-Par-1996-1.swf) are

Conversion 2 includes all the queue names that appear in the logs, which include a few that did not appear in the official documentation.
In conversion 2 CPU usage of 0 was changed to -1.

Regarding the OLD files SDSC-Par-1995-1.swf and SDSC-Par-1996-1.swf, on Dec 16, 2004 the following changes were made:

In SDSC-Par95, there were 11 jobs with wait-time smaller than -1 (jobs 17067, 17205, 31796, 31798, 39631, 39680, 43899, 43921, 44021, 47588, 49606), and 15 jobs with runtime smaller than -1 (jobs 17552, 17553, 17554, 17558, 17559, 17560, 17561, 17562, 17563, 17569, 17572, 17573, 17574, 17576, 17580). Most jobs with negative runtime also have unknown submission time (in the original log). In addition, negative wait-times (also from the original log) are too large to be caused due to unsynchronized clocks. For this reason, all the above were corrected to be -1 in the original SWF file. As for the clean SWF version: it contained only the 11 jobs with negative wait times; these were deleted.
Fixed large negative wait-time of 17 SDSC-PAR96 jobs to -1 (undefined). This was done to both SWF versions of the log (original and cleaned). Jobs are: 801, 1072, 1176, 6554, 6556, 6714, 6807, 9755, 9808, 14989, 15874, 25922, 25945, 25965, 25968, 33653, 37376

Usage Notes

These logs contains two types of anomalies that are not representative of normal usage. These have been removed in the cleaned versions of the logs, and it is recommended that these versions be used.
The cleaned logs are available as SDSC-Par-1995-3.1-cln.swf and SDSC-Par-1996-3.1-cln.swf.

The first anomaly is a set of 16 jobs that is executed every day at around 3:45 AM. These are probably automatic jobs that perform some system administration function. They were removed using the following filter:

(user=2 or user=5) and (submit_hour=3 or submit_hour=4)

In 1995, 6604 jobs were thus removed. In 1996, 5301 were removed.

The second anomaly is flurries of very high activity by individual users. Four flurries of different magnitudes occured in 1995. The filters used to remove them were

user=61 and job>17495 and job<25398 (2005 jobs)
user=62 and job>17441 and job<26321 (2139 jobs)
user=66 and job>55419 and job<69428 (8678 jobs)
user=92 and job>69856 and job<76815 (3476 jobs)

In addition, a single flurry occured in 1996. This was removed by the following filter

user=23 and job>3035 and job<4677 (1283 jobs)

Note that the filters were applied to the original logs, and unfiltered jobs remain untouched. As a result, in the filtered logs job numbering is not consecutive.

Further information on flurries and the justification for removing them can be found in:

D. G. Feitelson and D. Tsafrir, ``Workload sanitation for performance evaluation''. In IEEE Intl. Symp. Performance Analysis of Systems and Software, pp. 221-230, Mar 2006.
D. Tsafrir and D. G. Feitelson, ``Instability in parallel job scheduling simulation: the role of workload flurries''. In 20th Intl. Parallel and Distributed Processing Symp., Apr 2006.

The Log in Graphics

File SDSC-Par-1995-3.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance

File SDSC-Par-1995-3.1-cln.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance

File SDSC-Par-1996-3.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance

File SDSC-Par-1996-3.1-cln.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance

Parallel Workloads Archive - Logs