Experimental Methods in Computer Science

Exercise 10 – Experimenting with Users

Goals

• Appreciate the importance of experimenting with human subjects
• Understand the difficulties of experimenting with human subjects
• See how this works

Background

Humans are tightly involved with computer systems, either as the developers of a computer system or as the users of a computer system. Thus the behavior of humans has a direct impact on a system's design, usefulness, and usability.

In particular, system designers are often frustrated by users who "don't get it", and fail to appreciate their system's great features. Part of the problem is that the users may come from a different background and simply see things differently. Thus when designing a system it is extremely important to conduct experiments with real users. We will do a very small scale study of this type. The topic will be usability of menus in a desktop editing application.

Assignment

The experiment with the users will include two parts. The first is trying to accomplish several simple tasks. The second will be to provide insight into how the user sees things.

1. Tasks

   The tasks will be simple editing tasks from a desktop editing application such as PowerPoint (preparing a presentation). You should define about 5 simple tasks with increasing complexity. For example, the tasks can be as follows. Note that you need to define the tasks in a way that does not specify exactly how you want the user to perform them -- the user needs to figure this out.

   1. Create a slide that says "This is a TEST"
   2. Create a duplicate of this slide but with a different font type and size (you need to specify exactly which). Note that the definition is not "duplicate this slide", which implies that there is a "duplicate" function somewhere; we want to see whether users will look for such a function or just create a new slide and type the text again or use copy/paste.
   3. Delete the first slide
   4. Change the background of the slide to something (specify what)
   5. Put a picture from some file (specify) on the slide, and write the text "this is my picture" so that it partially overlaps the picture. Note that we don't say "insert a picture" which immediately points the user to the "insert" menu.

2. Insight.

   The goal here is to try and assess how the user sees the operations provided by the system. To do so, prepare some 25-40 cards with basic operations listed on them. Examples include

   1. create new slide
   2. duplicate current slide
   3. copy a slide from another presentation
   4. delete current slide
   5. delete a sequence of slides
   6. change font
   7. change font size
   8. change lettering color
   9. change background color
   and so on. mix the cards thoroughly. Then give them to the subject, and ask him or her to arrange them in groups that make sense to him or her. Emphasize the fact that there is no "correct" grouping. After this is done, ask him/her to label the groups.

   Given the grouping and labels, try to understand the common theme of each group. Check whether it corresponds to an existing menu or group of buttons, or rather crosses multiple menus/groups. If you're serious you can conduct a clustering analysis, where the strength of the connection between two cards is the number of times they are sorted together (but this is not a requirement).

Apart from designing the experiments outlined above, you need to conduct live experiments with several subjects. Real studies require a minimum of 5-6 subjects; if you can't find that many you are allowed to go down to 3 for this exercise. Note that your subjects should preferably not be expert computer users, so friends from other faculties or family (especially from older generations) make good subjects. In particular, it is imperative that they not be experienced with using the system you are testing. This implies that you should probably not use Microsoft Office, but rather use OpenOffice or Zoho.

Before you start, remember to tell your subjects that you are not testing them but rather testing the system they are about to use; there are no "right" or "wrong" answers, so they should freely do what they think is most reasonable at every instant. Importantly, they should say out loud what they are trying to do and why to allow you to follow. NEVER help your subjects or tell them what to do.

In a real setting, user experiments would be videotaped for later analysis. If you have the facilities you are invited to do so, but this is not a requirement for the exercise.

Submit

Submit a single pdf file that contains all the following information:

• Your names, logins, and IDs.
• Your subjects: how many you had, what was their background, and generally how well they performed.
• Results of the tasks experiment: what were your tasks, how did users perform them, and especially, were there any surprises.
• Results of the insight experiment: how did the subjects group your list of functions, and what you learn from that.

Please do the exercise in pairs.

To the course home page