(r3) HCIExperiments < Imager

Tags: view all tags

%TOC%

---+ HCI Experiments

A place to put tips, suggestions, and best practices on running experiments.

---++ Procedure
   1. Design and develop experiment
   1. Get ethics approval (http://www.rise.ubc.ca)
   1. Recruit participants
   1. Run the experiment
   1. Analyze the data
   1. Report the data

---++ Ethics

The following samples come from an accepted ethics application. It may be useful to reuse some of the boilerplate wording in future applications.

   * [[%ATTACHURL%/consent_form_ver1.1.docx][consent_form_ver1.1.docx]]: A sample consent form

   * [[%ATTACHURL%/recruitment_email_ver1.1.docx][recruitment_email_ver1.1.docx]]: A sample recruitment email

   * [[%ATTACHURL%/completed_ethics_application.pdf][completed_ethics_application.pdf]]: A sample completed ethics application



---++ Recruiting

There's a lot of great information on recruiting participants here: HCIStudyParticipantRecruitingResources.


---++ Running the Experiment

Don't forget to have an extra consent form for participants, should they want their own copy.

---+++ Booking a room
To book a room in the Usability Lab (X727), go to this site: http://hct.ece.ubc.ca/mrbs/admin.php. There, you'll need to create a new experiment before you can reserve a room for it. 

As of June 13, 2014, there is a javascript bug preventing easy creation of new experiments. When attempting to submit the new experiment form an error dialog pops up saying the Sponsor field is empty, despite it being filled in. If this happens, you can fix it by inspecting the Sponsor input field element in your browser and deleting it from the DOM. You'll note that there is an additional hidden input field there that still has the sponsor information, so the form should work correctly. The admins have been informed, but have decided not to fix it.


---+++ Recording the data

It's important to think about how you'll record the data before running the experiment. To run statistics on the results, you'll want to use a wide format for at least some of the data. By this I mean that each participant's data should be saved in a single row. It is a great idea to try and run some of your analysis on sample data from your experiment before running the experiment on real participants. This will hopefully ensure that you are recording all the data you need to analyze and storing it in a way that makes the analysis easy.

As an example, you might record data with the following format, where the top row of data is in the wide-format (for statistical analysis), summarized from the raw data below:

<verbatim>
userCorrect,disCorrect,dSlowDisCorrect,dMedDisCorrect,dFastDisCorrect,disOuterCorrect,disInnerCorrect,aCorrect,...
416,141,70,39,32,66,75,86,...

Block,Trial,UserCorrect,DistractorCorrect,UserSpeed,DistractorSpeed,DistractorRow,DistractorCol,...
slow,1,1,0,0,0,4,0,...
slow,2,1,0,0,1,2,0,...
slow,3,1,1,0,0,0,2,...
slow,4,1,0,0,2,2,1,...
slow,5,1,0,0,0,4,4,...
slow,6,1,0,0,2,3,3,...
slow,7,1,1,0,1,1,1,...
slow,8,1,0,0,2,2,3,...
...
</verbatim>

---+++ Participant receipt form

You'll need a form for people to sign off indicating that they have been paid by you for having done the experiment. A sample template to follow has been attached to this page: [[%ATTACHURL%/receipt_generic.docx][receipt_generic.docx]]. At the completion of the experiment, this can be passed on to the program assistant for reimbursement.


---++ Analyzing the Data

Common statistical methods used in analyzing experiment data.

---+++ Parametric tests
   * ANOVA
   * t-Test


---+++ Non-parametric Tests
For analyzing non-parametric data, such as Likert-formatted items, consider using the tests listed below. It is also worthwhile to read through [[%ATTACHURL%/brian_likert.pdf][an email from Brian Gleeson]] on the topic.

---++++ Repeated Measures / Within Subjects
   * Friedman Test
   * Wilcoxon Signed-Rank Test

---++++ Between Subjects
   * Kruskal-Wallis
   * Mann-Whitney U


---+++ Running the analysis in R

See examples on how to use R to analyze your data here: https://github.com/pbeshai/stats. Feel free to contribute to the repository.


---++ Reporting the data

TODO: Give examples of how to report the typical results. P-value and effect size. Examples for: ANOVA, t-test, Friedman, Wilcoxon, Kruskal-Wallis, and Mann-Whitney U.

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	brian_likert.pdf	r1	manage	275.5 K	2014-06-13 - 21:48	PeterBeshai	Email from Brian Gleeson on doing stats on Likert-formatted data
pdf	completed_ethics_application.pdf	r1	manage	862.8 K	2014-06-13 - 22:16	PeterBeshai	A sample completed ethics application
docx	consent_form_ver1.1.docx	r1	manage	25.9 K	2014-06-13 - 22:12	PeterBeshai	A sample consent form
docx	receipt_generic.docx	r1	manage	23.4 K	2014-06-13 - 21:42	PeterBeshai	Receipt sheet for participants to sign after doing the experiment
docx	recruitment_email_ver1.1.docx	r1	manage	17.9 K	2014-06-13 - 22:13	PeterBeshai	A sample recruitment email

Topic revision: r3 - 2014-06-16 - PeterBeshai