Classes are held on Tuesday 16-18 in Shprintzak 213
| meeting | material | exercises | |
|---|---|---|---|
| 1 | 15/2/11 |
Introduction:
(slides)
The experimental method: measurements, hypothesis testing, and reproducibility. Use of experimentation in engineering, performace evaluation, and algorithmics. Tichy, Should computer scientists experiment more? Computer, 1998. Feitelson, Experimental computer science: the need for a cultural change. Manuscript, 2005. |
Ex1: Data exploration and simple graphs |
| 2 | 22/2/11 |
Statistical graphics:
(slides)
Data visualization and exploring data. Representing experimental results graphically. Jain chap. 10, 11. Michael Friendly's Gallery of Data Visualization and Neoformix patterns in data. gnuplot reference, Ploticus reference. |
Ex2: Data presentation |
| 3 | 1/3/11 |
Measurement:
(slides)
Resolution, precision, and accuracy. Repeated measurements. Removing outliers. Confidence intervals. Jain chap 4, 5, 11 Lilja chap. 4, 6, 7 |
Ex3: Basic measurement |
| 4 | 8/3/11 |
Designing and using microbenchmarks. lmbench.
(slides)
Using the average, maximum, minimum, and median. Jain chap 12, 14 Lilja chap. 8 Staelin and McVoy, mhz microbenchmark Usenix, 1998 Ostrovsky on cache effects |
Ex4: Measuring cache parameters |
| 5 | 15/3/11 |
Data analysis
(slides)
Distribution fitting. Handling censored data. Correlation. Linear regression. Applying linear regression to transformed data. |
Ex5: Finding correlations in data |
| 6 | 22/3/11 |
Experiment design:
(slides)
Factors and their interactions. Simple experiment design, full factorial design. ANOVA. Jain chap. 16-19; Law/Kelton chap. 12 |
Ex6: 22 experiment design |
| 7 | 29/3/11 |
Workloads:
(slides)
Workload analysis and characterization, and data cleaning. Feitelson and Tsafrir, Workload sanitation for performance evaluation. IEEE Symp. Performance Analysis of Systems and Software, 2006 |
Ex7: What's wrong with this data? |
| 8 | 5/4/11 |
The arrival process: burstiness and cycles.
(slides)
Self-similarity. Correlation and locality of sampling. Paxson and Floyd, failure of Poisson modeling IEEE/ACM Trans. Networking, 1995 |
Ex8: Burstiness vs. Poisson |
| 9 | 12/4/11 |
Analyzing networks:
(guest lecture by Prof. Scott Kirkpatrick,
in English)
(slides)
From the Internet to telephone calls. |
Ex9: the graph of phone calls |
| 10 | 3/5/11 |
Experiments with users:
(slides)
Observing users, usability testing, interviews. |
Ex10: Usability of a GUI and menus |
| 11 | 17/5/11 |
Experimental algorithmics:
(slides)
what experiments can add to theory when studying algorithms |
Ex11: The complexity of sorting |
| 12 | 24/5/11 | Case studies of experimental evaluation of algorithms (slides) | Ex12: Comparing first-fit and best-fit |
| 13 | 31/5/11 |
Summary.
Experimentation and performance evaluation
(slides)
Final Quiz. |
|