HCIExperiments < Imager

Tags: view all tags

%TOC%

---+ HCI Experiments

A place to put tips, suggestions, and best practices on running experiments.

---++ Procedure
   1 Design and develop experiment
   1 Get ethics approval (http://www.rise.ubc.ca)
   1 Recruit participants
   1 Run the experiment
   1 Analyze the data
   1 Report the data

---++ Ethics

The following samples come from an accepted ethics application. It may be useful to reuse some of the boilerplate wording in future applications.

   * [[%ATTACHURL%/consent_form_ver1.1.docx][consent_form_ver1.1.docx]]: A sample consent form

   * [[%ATTACHURL%/recruitment_email_ver1.1.docx][recruitment_email_ver1.1.docx]]: A sample recruitment email

   * [[%ATTACHURL%/completed_ethics_application.pdf][completed_ethics_application.pdf]]: A sample completed ethics application

---++ Recruiting

There's a lot of great information on recruiting participants here: HCIStudyParticipantRecruitingResources.

---++ Running the Experiment

Don't forget to have an extra consent form for participants, should they want their own copy.

---+++ Booking a room

To book a room in the Usability Lab (X727), go to this site: <a href="https://mrbs.icics.ubc.ca/" target="_blank">https://mrbs.icics.ubc.ca</a>. There, you'll need to create a new experiment before you can reserve a room for it.

As of June 13, 2014, there is a javascript bug preventing easy creation of new experiments. When attempting to submit the new experiment form an error dialog pops up saying the Sponsor field is empty, despite it being filled in. If this happens, you can fix it by inspecting the Sponsor input field element in your browser and deleting it from the DOM. You'll note that there is an additional hidden input field there that still has the sponsor information, so the form should work correctly. The admins have been informed, but have decided not to fix it. (As of August 2015, the stupid bug is still there, and the website is still terrible. For instance, beware that the "end date" will not be included if you create a daily repeat booking).

---+++ Recording the data

It's important to think about how you'll record the data before running the experiment. To run statistics on the results, you'll want to use a wide format for at least some of the data. By this I mean that each participant's data should be saved in a single row. It is a great idea to try and run some of your analysis on sample data from your experiment before running the experiment on real participants. This will hopefully ensure that you are recording all the data you need to analyze and storing it in a way that makes the analysis easy.

As an example, you might record data with the following format, where the top row of data is in the wide-format (for statistical analysis), summarized from the raw data below:
<verbatim>
userCorrect,disCorrect,dSlowDisCorrect,dMedDisCorrect,dFastDisCorrect,disOuterCorrect,disInnerCorrect,aCorrect,...
416,141,70,39,32,66,75,86,...

Block,Trial,UserCorrect,DistractorCorrect,UserSpeed,DistractorSpeed,DistractorRow,DistractorCol,...
slow,1,1,0,0,0,4,0,...
slow,2,1,0,0,1,2,0,...
slow,3,1,1,0,0,0,2,...
slow,4,1,0,0,2,2,1,...
slow,5,1,0,0,0,4,4,...
slow,6,1,0,0,2,3,3,...
slow,7,1,1,0,1,1,1,...
slow,8,1,0,0,2,2,3,...
...</verbatim>

---+++ Participant receipt form

You'll need a form for people to sign off indicating that they have been paid by you for having done the experiment. A sample template to follow has been attached to this page: [[%ATTACHURL%/receipt_generic.docx][receipt_generic.docx]]. At the completion of the experiment, this can be passed on to the program assistant for reimbursement.

---++ Analyzing the Data

Common statistical methods used in analyzing experiment data.

---+++ Parametric tests
   * ANOVA
   * t-Test

Note that as per Cohen (1988), eta squared values map to the following effect sizes:

   * .01 = small
   * .06 = medium
   * .14 = large
   * Jacob Cohen (1988), Statistical Power Analysis for the Behavioral Sciences (2nd ed.), New Jersey: Lawrence Erlbaum Associates, ISBN 0-8058-0283-5.

---+++ Non-parametric Tests

For analyzing non-parametric data, such as Likert-formatted items, consider using the tests listed below. It is also worthwhile to read through [[%ATTACHURL%/brian_likert.pdf][an email from Brian Gleeson]] on the topic.

---++++ Repeated Measures / Within Subjects
   * Friedman Test
   * Wilcoxon Signed-Rank Test

---++++ Between Subjects
   * Kruskal-Wallis
   * Mann-Whitney U

---+++ Running the analysis in R

See examples on how to use R to analyze your data here: https://github.com/pbeshai/stats. Feel free to contribute to the repository.

You may also find examples and explanation at this page by Microsoft Research's Koji Yatani helpful: http://yatani.jp/HCIstats/HomePage.

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	brian_likert.pdf	r1	manage	275.5 K	2014-06-13 - 21:48	PeterBeshai	Email from Brian Gleeson on doing stats on Likert-formatted data
pdf	completed_ethics_application.pdf	r1	manage	862.8 K	2014-06-13 - 22:16	PeterBeshai	A sample completed ethics application
docx	consent_form_ver1.1.docx	r1	manage	25.9 K	2014-06-13 - 22:12	PeterBeshai	A sample consent form
docx	receipt_generic.docx	r1	manage	23.4 K	2014-06-13 - 21:42	PeterBeshai	Receipt sheet for participants to sign after doing the experiment
docx	recruitment_email_ver1.1.docx	r1	manage	17.9 K	2014-06-13 - 22:13	PeterBeshai	A sample recruitment email

Raw edit | More topic actions

Topic revision: r8 - 2015-08-21 - aponsard