A place to put tips, suggestions, and best practices on running experiments.
The following samples come from an accepted ethics application. It may be useful to reuse some of the boilerplate wording in future applications.
There's a lot of great information on recruiting participants here: HCIStudyParticipantRecruitingResources.
Don't forget to have an extra consent form for participants, should they want their own copy.
As of June 13, 2014, there is a javascript bug preventing easy creation of new experiments. When attempting to submit the new experiment form an error dialog pops up saying the Sponsor field is empty, despite it being filled in. If this happens, you can fix it by inspecting the Sponsor input field element in your browser and deleting it from the DOM. You'll note that there is an additional hidden input field there that still has the sponsor information, so the form should work correctly. The admins have been informed, but have decided not to fix it.
It's important to think about how you'll record the data before running the experiment. To run statistics on the results, you'll want to use a wide format for at least some of the data. By this I mean that each participant's data should be saved in a single row. It is a great idea to try and run some of your analysis on sample data from your experiment before running the experiment on real participants. This will hopefully ensure that you are recording all the data you need to analyze and storing it in a way that makes the analysis easy.
As an example, you might record data with the following format, where the top row of data is in the wide-format (for statistical analysis), summarized from the raw data below:
userCorrect,disCorrect,dSlowDisCorrect,dMedDisCorrect,dFastDisCorrect,disOuterCorrect,disInnerCorrect,aCorrect,... 416,141,70,39,32,66,75,86,... Block,Trial,UserCorrect,DistractorCorrect,UserSpeed,DistractorSpeed,DistractorRow,DistractorCol,... slow,1,1,0,0,0,4,0,... slow,2,1,0,0,1,2,0,... slow,3,1,1,0,0,0,2,... slow,4,1,0,0,2,2,1,... slow,5,1,0,0,0,4,4,... slow,6,1,0,0,2,3,3,... slow,7,1,1,0,1,1,1,... slow,8,1,0,0,2,2,3,... ...
You'll need a form for people to sign off indicating that they have been paid by you for having done the experiment. A sample template to follow has been attached to this page: receipt_generic.docx. At the completion of the experiment, this can be passed on to the program assistant for reimbursement.
Common statistical methods used in analyzing experiment data.
Note that as per Cohen (1988), eta squared values map to the following effect sizes:
See examples on how to use R to analyze your data here: https://github.com/pbeshai/stats. Feel free to contribute to the repository.
You may also find examples and explanation at this page by Microsoft Research's Koji Yatani helpful: http://yatani.jp/HCIstats/HomePage.
I | Attachment | History | Action | Size | Date | Who | Comment |
---|---|---|---|---|---|---|---|
![]() |
brian_likert.pdf | r1 | manage | 275.5 K | 2014-06-13 - 21:48 | PeterBeshai | Email from Brian Gleeson on doing stats on Likert-formatted data |
![]() |
completed_ethics_application.pdf | r1 | manage | 862.8 K | 2014-06-13 - 22:16 | PeterBeshai | A sample completed ethics application |
![]() |
consent_form_ver1.1.docx | r1 | manage | 25.9 K | 2014-06-13 - 22:12 | PeterBeshai | A sample consent form |
![]() |
receipt_generic.docx | r1 | manage | 23.4 K | 2014-06-13 - 21:42 | PeterBeshai | Receipt sheet for participants to sign after doing the experiment |
![]() |
recruitment_email_ver1.1.docx | r1 | manage | 17.9 K | 2014-06-13 - 22:13 | PeterBeshai | A sample recruitment email |