HCI Experiments

A place to put tips, suggestions, and best practices on running experiments.

Procedure

1. Design and develop experiment
2. Get ethics approval (http://www.rise.ubc.ca)
3. Recruit participants
4. Run the experiment
5. Analyze the data
6. Report the data

Ethics

The following samples come from an accepted ethics application. It may be useful to reuse some of the boilerplate wording in future applications.

• consent_form_ver1.1.docx: A sample consent form

• recruitment_email_ver1.1.docx: A sample recruitment email

• completed_ethics_application.pdf: A sample completed ethics application

Recruiting

There's a lot of great information on recruiting participants here: HCIStudyParticipantRecruitingResources.

Running the Experiment

Don't forget to have an extra consent form for participants, should they want their own copy.

Booking a room

To book a room in the Usability Lab (X727), go to this site: http://hct.ece.ubc.ca/mrbs/admin.php. There, you'll need to create a new experiment before you can reserve a room for it.

As of June 13, 2014, there is a javascript bug preventing easy creation of new experiments. When attempting to submit the new experiment form an error dialog pops up saying the Sponsor field is empty, despite it being filled in. If this happens, you can fix it by inspecting the Sponsor input field element in your browser and deleting it from the DOM. You'll note that there is an additional hidden input field there that still has the sponsor information, so the form should work correctly. The admins have been informed, but have decided not to fix it.

Recording the data

It's important to think about how you'll record the data before running the experiment. To run statistics on the results, you'll want to use a wide format for at least some of the data. By this I mean that each participant's data should be saved in a single row. It is a great idea to try and run some of your analysis on sample data from your experiment before running the experiment on real participants. This will hopefully ensure that you are recording all the data you need to analyze and storing it in a way that makes the analysis easy.

As an example, you might record data with the following format, where the top row of data is in the wide-format (for statistical analysis), summarized from the raw data below:

userCorrect,disCorrect,dSlowDisCorrect,dMedDisCorrect,dFastDisCorrect,disOuterCorrect,disInnerCorrect,aCorrect,...
416,141,70,39,32,66,75,86,...

Block,Trial,UserCorrect,DistractorCorrect,UserSpeed,DistractorSpeed,DistractorRow,DistractorCol,...
slow,1,1,0,0,0,4,0,...
slow,2,1,0,0,1,2,0,...
slow,3,1,1,0,0,0,2,...
slow,4,1,0,0,2,2,1,...
slow,5,1,0,0,0,4,4,...
slow,6,1,0,0,2,3,3,...
slow,7,1,1,0,1,1,1,...
slow,8,1,0,0,2,2,3,...
...

Participant receipt form

You'll need a form for people to sign off indicating that they have been paid by you for having done the experiment. A sample template to follow has been attached to this page: receipt_generic.docx. At the completion of the experiment, this can be passed on to the program assistant for reimbursement.

Analyzing the Data

Common statistical methods used in analyzing experiment data.

Parametric tests

• ANOVA
• t-Test

Note that as per Cohen (1988), eta squared values map to the following effect sizes:

• .01 = small
• .06 = medium
• .14 = large
• Jacob Cohen (1988), Statistical Power Analysis for the Behavioral Sciences (2nd ed.), New Jersey: Lawrence Erlbaum Associates, ISBN 0-8058-0283-5.

Non-parametric Tests

For analyzing non-parametric data, such as Likert-formatted items, consider using the tests listed below. It is also worthwhile to read through an email from Brian Gleeson on the topic.

Repeated Measures / Within Subjects

• Friedman Test
• Wilcoxon Signed-Rank Test

Between Subjects

• Kruskal-Wallis
• Mann-Whitney U

Running the analysis in R

See examples on how to use R to analyze your data here: https://github.com/pbeshai/stats. Feel free to contribute to the repository.

You may also find examples and explanation at this page by Microsoft Research's Koji Yatani helpful: http://yatani.jp/HCIstats/HomePage.