Performance Evaluation Links
Professional Societies
Journals
Conferences
General
- ACM SIGMETRICS International Conference on Measurement and
Modeling of Computer Systems.
Archive
in ACM digital library.
Web site for current year linked from the
SIGMETRICS home page.
- PERFORMANCE: IFIP WG 7.3 International Symposium on Computer Performance
Modeling, Measurement and Evaluation.
Held in conjunction with SIGMETRICS every third year, and independently
in between.
Proceedings published as special issues of Elsevier's
Performance
Evaluation journal:
1992 is vol. 16;
1996 is vol. 27-28;
1999 is vol. 36-37;
2002 is vol. 49;
2005 is vol. 62;
2007 is vol. 64 no. 9-12;
2010 is vol. 67 no. 11;
2011 is vol. 68 no 11;
- MASCOTS:
the International Symposium on Modeling, Analysis and Simulation of
Computer and Telecommunication Systems.
Archive
in the IEEE CS digital library, or
in IEEE Xplore.
- WSC: Winter Simulation Conference.
Archive
in ACM digital library or from
INFORMS
Simulation Society, including tutorials.
- MAMA: Workshops on
MAthematical performance Modeling and Analysis co-located
with SIGMETRICS.
- QEST: International Conference on Quantitative
Evaluation of SysTems.
Networking and Internet
- IMC: Internet Measurement Conference.
- PAM: Passive and Active Measurement
Conference.
Archive
in Springer Link.
- International Teletraffic Congress.
Emphasis on Architecture
- ISPASS: IEEE International Symposium on
Performance Analysis of Systems and Software.
Archive
in IEEE Xplore.
- IISWC: IEEE International Symposium
on Workload Characterization,
(previously called WWC:
Workshop on Workload Characterization).
Archive
in IEEE Xplore.
- MoBS: Workshop
on Modeling, Benchmarking and Simulation
- PMBS: Performance Modeling,
Benchmarking and Simulation of High Performance Computer Systems
- MoBS: Workshop
on Modeling, Benchmarking and Simulation
- CAECW:
Workshops on Computer Architecture Evaluation using Commercial Workloads
(discontinued?)
Other
- ICPE: International Conference
on Performance Engineering.
Archive in ACM digital library or
at SPEC Research Group.
- WOSP: International Workshop on Software and Performance
(subsumed in ICPE since 2010).
Archive
in ACM digital library.
- SPECTS: International Symposium on Performance Evaluation of
Computer and Telecommunication Systems.
Web sites for
2003,
2004,
2005,
2006,
2007,
2008,
2009,
2010,
2011,
2012,
2013.
- Workshop on Experimental Computer
Science
- Workshop on Analysis of
System Logs
- JSSPP: Workshops on
Job Scheduling Strategies for Parallel Processing.
Data (Traces and Logs)
Benchmarks
Architecture
- SPEC: organization defining
benchmarks, notably for CPU performance.
- Listing
of benchmarks from the WWW Computer Architecture Page
at Wisconsin.
- FreeBench: an attempt to
define an open-source alternative to SPEC.
- lmbench:
characterizing the performance of a computing platform (processor,
memory, etc.)
- Stream:
sustainable memory bandwidth in high performance computers.
Parallel and HPC
- HPC Challenge Benchmark:
a multi-dimensional benchmark for high-performance systems (supercomputers)
intended to provide a better characterization than Linpack.
- SPLASH:
Stanford Parallel Applications for Shared Memory.
- PARSEC:
a multi-threaded benchmark for CMPs.
- ParkBench:
benchmark for parallel systems.
- NAS Parallel Benchmarks
for evaluation of parallel supercomputers.
- ASCI
Purple benchmarks defined as part of the RFP.
Networking
- NetPerf:
benchmark and results for network performance.
- coNCePTuaL:
a high-level language to define communication tests and benchmarks.
- Effective bandwidth benchmark.
- IETF
Benchmarking Methodology Working Group.
Definitions and procedures for benchmarking.
Storage and I/O
- IOzone Filesystem Benchmark.
- FileBench.
- Filesystem and storage
benchmarking portal.
- AC&NC I/O
benchmarks page, with links to many I/O benchmarks for Unix, DOS,
and Windows.
Other
- contest:
the Linux kernel responsiveness benchmark.
- BioBench/BioParallel:
A Benchmark Suite for Bioinformatics Applications.
- The DaCapo benchmark
suite for Java benchmarking.
- MiBench, a
benchmark for embedded systems.
Also has a set of
alternative inputs to allow for more representative evaluations
and characterize performance variability.
- The TPC benchmark.
- Compilation
of industry benchmark results.
- The
Benchmark Handbook for Database and Transaction Processing Systems,
Jim Gray (ed.), Morgan Kaufmann, 2nd Edition, 1993.
- The MediaEval
Benchmark: A series of workshops using predefined tasks and
datasets to evaluate multimedia-related applications.
Software
Monitoring
- The Linux Trace Toolkit.
- Klogger:
a logging framework for the Linux kernel.
- PAPI: performance API, providing
a common API to access performance counters on diverse architectures.
- Paradyn parallel
performance tools from U Wisconsin -- dynamic instrumentation of
programs.
- Pin: dynamic
instrumentation on Intel platforms.
- Monarch: emulating and
measuring TCP flows.
- Dimes: agent for mapping the
Internet topology.
- Pablo
performance analysis toolkit from U Illinois.
Simulation
- DiskSim disk system
simulator.
- ns-2 network simulator.
- Listing of Internet
traffic generators.
- Listing
of simulation software from the WWW Computer Architecture Page
at Wisconsin.
- Listing
of simulation software from the simulation education web site.
See also their page on
selecting
simulation software.
Data analysis and visualization
- NIST
Dataplot,
software accompanying the statistics handbook.
- The R project, an
open-source counterpart of S, for statistical computing and graphics.
- GGobi, a graphical multivariate data
exploration tool.
- Ploticus
data display engine.
- Gnuplot
data and function plotting engine.
- ViSta visual statistics system
- |STAT data manipulation and
analysis programs by Gary Perlman.
- Various
tools for variate creation, simulation, etc. by Ken Christensen.
- ZunZun online curve and surface fitting.
- ExpertFit
distribution fitting software.
- Various software from Palisade, with a
bias towards business applications, decision support, etc.
Includes BestFit
distribution fitting software
- S-PLUS data analysis and
visualization.
- many eyes:
interactive visualization and brainstorming.
Other
- Auto-pilot:
tools for running benchmarks.
- Mersenne
Twister random number generator.
- Design-Expert 6 from Stat-Ease
for experimental design and analysis.
- StatPages.net: Links to
lots of on-line statistics resources, especially for doing various
calculations and applying various tests.
- Statistics.Com: Links to
lots of on-line statistics resources, including software.
Physical Infrastructure
Educational Material
- Listing of
queueing theory resources, including FREE books and course
notes, collected by Myron Hlynka.
- Lecture
notes (actually a book) on performance evaluation by Jean-Yves Le
Boudec.
- Quantitative
System Performance: the classic text by Lazowska, Zahorjan,
Graham, and Sevcik.
- Projects on experimentation in
computer science by Steve Robbins.
- Annotated
bibliography on self similarity.
- Simulation
Education HomePage by Bill Yurcik.
An amazing web page, with a wealth of resources for teaching/learning
about simulation (including software, books, links to courses, etc.).
Regrettably not being updated any more.
- Zipf's Law web site.
- The
queueing theory tutor with cute animation.
- Michael Friendly's Gallery of Data
Visualization.
Statistics
- The
Engineering Statistics Handbook from NIST, with chapters on data analysis,
measurement, modeling, etc.
- Electronic
Textbook on Statistics from StatSoft.
- Visual Statistics, with
detailed tutorials.
- Computational
statistics.
- American
statistical association online resources and links.
- Second Moment -- applied
statistics site.
- Statistical science resources.
- Practical
insights on statistics by Charles Annis, P.E.
- David Lane's HyperStat
online book.