Experimental Methods in Computer Science
Exercise 6 – Experimental Design
Goals
- Use and understand experimental design and ANOVA
- Gain some more experience with simple measurements
Assignment
In this exercise we will use a 22 experiment design to
assess the performance impact of two factors.
Specifically, the time to call a function depends on two factors:
- The number of arguments
- The size each argument
Using two levels for each of the factors, and conducting a full factorial
design, means measuring four combinations.
The analysis will then tell us which factor has a larger effect: the number of
arguments, their size, or maybe the combination of the two.
Can you guess what the result will be?
Assignment
- Run experiments
Measure the time to call an empty function with either 2 or 10 arguments, each
of which is either an int or a double.
If this bores you, you can use other numbers or sizes; it would be interesting
to see how this affects the conclusions.
Remember to do the measurements carefully in order to obtain reliable results.
- Analyze the results
First, use a set of equations directly or a sign table to find the
effect of each factor.
The factors are the number of arguments and their size, represented by the
following two variables:
| variable | value | meaning |
| xA | -1 | int |
| 1 | double |
| xB | -1 | 2 arguments |
| 1 | 10 arguments |
The performance model is
y = q0 +
qA xA +
qB xB +
qAB xA xB
The four measurements with 4 combinations of size and number
give 4 values of y, yielding 4 equations from which you can find the
four q's.
In case you don't remember, the sign table is
| I | A | B | AB | y |
| 1 | -1 | -1 | 1 | |
| 1 | 1 | -1 | -1 | |
| 1 | -1 | 1 | -1 | |
| 1 | 1 | 1 | 1 | |
- Repeat this for the multiplicative model, i.e. using the log of
the results.
Submit
Submit
a single pdf file that contains all the following information:
- Your names, logins, and IDs
- Specifics of where you ran the measurements: what machine, it's
clock speed.
- A short description of how you conducted the measurements.
- Your raw measurement results.
- Your analysis and its conclusions; specifically, does the function call
time depend more on the number of arguments, their size, or the
interaction between these factors?
- The results, analysis, and conclusions for the multiplicative model
- Answer the question: which model is more reasonable in this case?
- The program you wrote to perform the measurements.
Submission deadline is
Monday morning, 28/3/11,
so I can give feedback in class on Tuesday.
To the course home page