Parallel Workloads Archive: Intel Netbatch

The Intel Netbatch logs

System:	Intel Netbatch Grid
Duration:	November 2012
Jobs:	48,821,850

These logs contains one month's worth of accounting records from the Intel Netbatch grid. This grid is composed of multiple clusters (called physical pools) in different locations around the world, each with tens of thousands of nodes. The data comes from four such pools, three on the west coast in the US and one in Israel.

The Intel Netbatch workload logs were graciously provided by Ohad Shai, Edi Shmueli, and Nir Antebi from Intel. If you use this log in your work, please use a similar acknowledgment. You can also cite the following paper where they were introduced:

Ohad Shai, Edi Shmueli, and Dror G. Feitelson, “Heuristics for resource matching in Intel's compute farm”. In Job Scheduling Strategies for Parallel Processing, Walfredo Cirne and Narayan Desai, (ed.), Springer-Verlag, 2013.

Downloads:

Intel-NetbatchA-2012-0	592 MB gz	original log
Intel-NetbatchB-2012-0	475 MB gz	original log
Intel-NetbatchC-2012-0	602 MB gz	original log
Intel-NetbatchD-2012-0	422 MB gz	original log
Intel-NetbatchA-2012-1	142 MB gz	converted log
Intel-NetbatchB-2012-1	139 MB gz	converted log
Intel-NetbatchC-2012-1	126 MB gz	converted log
Intel-NetbatchD-2012-1	93 MB gz	converted log

(May need to click with right mouse button to save to disk)

Papers Using this Log:

This log was used in the following papers:
[shai13] [wang18]

System Environment

The Intel Netbatch grid is composed of multiple physical pools with different numbers of nodes, typically tens of thousands. Exact details have not been disclosed.

The vast majority of jobs are serial. However, there are also some parallel MPI jobs. Scheduling is done by the in-house Netbatch system.

Log Format

The original logs are available as Intel-NetbatchX-2012-0 (where "X" is A, B, C, or D for the four different pools).

These files are in CSV format. They contain one line per completed job with the following comma-separated fields:

job ID (anonymized)
group (anonymized)
user (anonymized)
command (anonymized)
submit time (local timestamp)
start time (local timestamp)
end time (local timestamp)
exit status
suspend time (seconds)
wall time (seconds)
user CPU time (seconds total for all threads)
system CPU time (seconds total for all threads)
max RSS (4KB pages)
max VM (4KB pages)
machine ID (anonymized)
iteration number
iteration submit time (local timestamp)
memory (GB total for all threads)
cores

Conversion Notes

The converted logs are available as Intel-NetbatchX-2012-1 (where "X" is A, B, C, or D for the four different pools). The conversion from the original format to SWF was done subject to the following.

The number of cores used was sometimes non-integral, reflecting time slicing on the cores being used. This happened only on pool A, for 158,440 jobs. This was rounded up to the nearest integer.
The status 0 was taken to mean success, and was converted to 1. All other status values were converted to 0.
User time was added to system time to derive CPU time.
Iterations of the same job were taken to imply dependencies. If a job had multiple iterations, they were sorted according to the iteration number, and each one was made dependent on the previous one. The think times were then computed as the difference between the end time of one iteration and the submit time of the next one.
The conversion loses the following data, that cannot be represented in the SWF:
- The details about maximal RSS and VM usage.
- The time that each job was suspended, if any.
The following anomalies were identified in the conversion of the pool A log:
- 530,291 jobs had an average CPU time higher than their runtime. In 239,094 cases the extra CPU time was larger than 1 minute.
- 3,591,888 jobs were recorded as using 0 CPU time; this was changed to -1. Of these jobs, 771,945 had "success" status.
- Negative run times occured 5 times; they were changed to 0
- Negative wait times occured 1944 times; they were changed to 0
- Non-integral number of cores occured 158,440 times. These were rounded up. Most were 0.5 and were rounded to 1.
- Iteration numbers were non-consecutive in 138,153 cases.
The following anomalies were identified in the conversion of the pool B log:
- 457,110 jobs had an average CPU time higher than their runtime. In 280,902 cases the extra CPU time was larger than 1 minute.
- 6,846,954 jobs were recorded as using 0 CPU time; this was changed to -1. Of these jobs, 286,620 had "success" status.
- Negative run times occured 1002 times; they were changed to 0. In 976 cases the negative run times were larger than 1 minute.
- Negative wait times occured 241 times; they were changed to 0. In 180 cases the negative wait times were larger than 1 minute.
- Iteration numbers were non-consecutive in 5,690 cases.
The following anomalies were identified in the conversion of the pool C log:
- 2,160,313 jobs had an average CPU time higher than their runtime. In 100,712 cases the extra CPU time was larger than 1 minute.
- 1,036,322 jobs were recorded as using 0 CPU time; this was changed to -1. Of these jobs, 448,697 had "success" status.
- Negative run times occured 257 times; they were changed to 0. In 255 cases negative run times were larger than 1 minute.
- Negative wait times occured 52 times; they were changed to 0. In 52 cases negative wait times were larger than 1 minute.
- Iteration numbers were non-consecutive in 2,880 cases.
The following anomalies were identified in the conversion of the pool D log:
- 261,080 jobs had an average CPU time higher than their runtime. In 131,392 cases the extra CPU time was larger than 1 minute.
- 1,328,197 jobs were recorded as using 0 CPU time; this was changed to -1. Of these jobs, 454,094 had "success" status.
- Negative run times occured 730 times; they were changed to 0. In 715 cases negative run times were larger than 1 minute.
- Negative wait times occured 312 times; they were changed to 0. In 312 cases negative wait times were larger than 1 minute.
- Iteration numbers were non-consecutive in 1,536 cases.

The conversion was done by a log-specific parser in conjunction with a more general converter module.

Usage Notes

The log from pool D appears to have a couple of flurries. In other pools some users are much more active than others, but this is not concentrated in a short time span.

The Logs in Graphics

Note: the zig-zag pattern in the utilization and offered load graphs is an artifact of showing only the maximal and minimal values for each day. The utilization is the number (and not the fraction) of concurrently allocated cores.

File Intel-NetbatchA-2012-1.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load