Tags: view all tags

C-TOC Usability Evaluation - Iteration 1

Interview 1 (subject #1)

Details

Date/time: Monday, May 17, 2010, 10am - 12:30pm
Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
Interviewers: Claudia (clinical), Matt (HCI)

Interviewee:
- age/gender: 76 / M
- profession: physics professor (retired) (still gives guest lectures)
- diagnosis: NCI (has memory complaints, esp. for remembering names)
- other: cataracts, poor eyesight
- computer literacy: atypically high
  - has used a Mac for research since early 80's (views as research tool only)
  - currently owns 28" iMac at home, another 24" iMac at the office, and a laptop;
  - has only ever used a one-button Mac mouse (left-handed)
  - uses Adobe Illustrator to create figures for academic publications / presentations
  - uses Google as research tool
  - uses MS Word, email, safari web browser, spreadsheets
  - no games, frivolous web browsing (viewed as waste of time)
  - self-taught computer user (skill transfer from type-writer)
  - relies on friends / grandsons for resolving computer issues; doesn't rely on help documentation

Notes

smaller font size on instruction screens
intro. checklist too much
show task completion / progress (i.e. "slide x of y")
break up sentence clauses on to new lines
PPT slideshow semitransparent navigation buttons are misleading and conflict w/ C-TOC navigation buttons
wording disliked: "battery", "practice"
why is the test / task necessary? what does it assess? otherwise it is viewed as frivolous nonsense
tests are mostly fun/easy
unaware of implied instructions to remember word/picture pairs
sentence comprehension: difficulty remembering instructions
trails: requires 2 reads of instructions
misplaced objects recall: poor performance

afterwards, eye strain, fatigues (2.5 hours)
better to have examiner present; would not want to take online
test is too long (known), interacting w/ computers seen as artificial, wouldn't be as fun alone, seeming like a chore

test fixes:

symbol-digit matching: clicking on number fields not working, only clicking on numbers
similarities: instructions at top of test
sentence comprehension: instructions should be on screen
sentence production: mechanically moving words not functional
go-stop - slides not advancing; useless

not yet interactive / implemented

sentence comprehension
sentence production
pattern construction
pattern recall
square puzzles
go-stop

Interview 2 (subject #2)

Details

Date/time: Thursday, June 3rd, 2010, 10am - 12:30pm
Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
Interviewers: Claudia (clinical), Matt (HCI)

Interviewee:
- age/gender: ?? / M
- profession: former VP-finance of large corporation (retired)
- diagnosis: MCI - mild AD - short term memory impairment evident (MMSE = 23)
- other: wears glasses
- computer literacy: some experience (modest)
  - first used a computer ~20 years ago at work; took classes - used for interpretation of results, reading documents, prepared financial statements, business correspondence, recalls using Lotus
  - currently owns a desktop PC running windows, claims to be 'archaic', which he bought himself
  - right-handed, familiar with a mouse
  - ~10 hours a week of computer use/week (generous), mostly games (FreeCell and Scrabble); not the highlight of his week, but some enjoyment is had
  - occasionally responds to email (ranging from 1/month to 1/week)
  - doesn't use computers elsewhere
  - familiar with MS Word
  - doesn't use web browsers mentioned (IE / Firefox - confused about question)
  - doesn't rely on help documentation

Notes

intro: first slide: missed bubble in bottom right to proceed to next slide; clicked anywhere expecting to proceed
current date on temporal orientation test: May 18, spring begins on May 28?
symbol-digit matching: unsure initially how to proceed
memory tasks (word recognition, misplaced object recall, pattern recall) significantly impaired, sentence comprehension suffered due to memory impairments
pattern construction: square inside triangle corners result was odd and very spaced-out compared to target
arithmetic appropriate for his background (finance), likely challenging for most others
mouse drag-and-drop habit hard to shake at times - would likely need to adopt traditional drag-and-drop formalism
square puzzles: perhaps add a counter, keeping track of the # of lines already moved
go-stop test purpose not clear: "am I supposed to do something?"
reflection: enjoyed challenges, unsure about accessing on the web (would need help from companion), can't recall help menu or what he would add to one, would need help setting up the test

test fixes:

technical observations and/or suggested quick C-TOC fixes:
- throughout: next/prev buttons on all slides for consistency (except slides in which it is states "click anywhere to continue")
- symbol-digit matching test: instructions to click on response bar need to be clearer; there may be a confusion with the source bar
- pattern construction test: instructions on where to pick up shapes and where to drop shapes need to be clearer; potential confusion resulting from attempting to pick up pieces of the target shape;
- trails test: it is not clear that clicking is not required in this test
- sentence comprehension test: too many test items;
  - mouse glitches resulting from drag operation (conflicts with the point-and-click move macro) - we identified this problem at Douglas College;
  - 'smallest to largest' cue used in one trial not visible enough
- misplaced object search test: include a 'give up' or 'I don't know' button to proceed to next trial
- sentence production test: slide advancement on first 'click anywhere to begin' slide glitches (likely a memory + PPT issue);
  - mouse glitches resulting from drag operation - specifically dropping problems (conflicts with the point-and-click move macro) - we identified this problem at Douglas College;
  - still missing a 'use as many words as possible' instruction - were we planning on adding this after our meeting with Barbara?
- square puzzles test: practice trial is not given instructions, but test trials are given instructions;
  - presently difficult to grab lines (another hassle of the point-and-click move macro) - could we widen the lines slightly to facilitate grabbing? would this have any impact on the results? (i.e. from 1pt to 3pt)
- go-stop test: some slide advancement problems (likely a memory + PPT issue again)
- misplaced-object recall test: instructions not clear w.r.t. the object being misplaced vs. contextually appropriate?;
  - some slide advancement problems (likely a memory + PPT issue again)

Interview 3

Details

Date/time: Friday, June 4th, 2010, 10am - 12:30pm
Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
Interviewers: Claudia (clinical), Matt (HCI)

Interviewee:
- age/gender: 71 / F
- profession: retired medical transcriber
- diagnosis: mild AD + on anti-psychotic medication to alleviate delirium (MMSE = 20)
  - problems with attention, balance, hearing
- other: wears glasses
- education: Capilano college, Political science
- computer literacy: low
  - an experienced typist, developed CT syndrome
  - did use a computer many years ago, floppy disk, transcriptions for doctors, inkjet printer
  - learned computer use through a combination of means
  - has a computer at home in disuse ("in limbo" - email full, automatic updates a hassle)
  - not familiar with a mouse, no longer uses a computer at all

Notes:

withdrew after 2 tests (picture-word pairs test and word-recognition test)
would need companion present to take test / take in a controlled lab setting

test fixes:

technical observations and/or suggested quick C-TOC fixes:
- make nav. buttons larger
- add "see prev. screen" to help menu where available/possible
- picture-word pairs test: instructions to click/advance needed on all slides where currently missing
  - some ambiguous / non-prototypical object images
- word recognition test: "click on numbered choice" instruction too small, to be moved to top of screen

Interview 4 (subject #3)

Details

Date/time: Thursday, June 17, 2010, 10am - 12:30pm
Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #8
Interviewers: Claudia (clinical), Matt (HCI)

Interviewee:
- age/gender: 68 / F
- profession: retired nurse / hospital shift supervisor
- diagnosis: FTD (early stage)
  - problems with distraction/attention, balance, tingling in extremities (fingers/toes - likely related to Diabetes)
- other: wears glasses - early cataracts - otherwise fine
- education: B.Sc nursing
- computer literacy: somewhat - not much use of computers currently (no longer needs to use most applications) - seen as very useful, quite enjoys using a computer
  - experienced due to work-related computer use; retrieving patient records, scheduling, biomedical services, organising staffing - once offered a manager position in which she would teach computer use skills to other staff members at Kewlowna Gen. Hospital
  - now: reads emails infrequently, sometimes will send an email, views photos sent to her and her husband; husband helps with computer use
  - not familiar with majority of computer programs - no web browsing, document writing, at least not recently
  - once borrowed her daughter's Wii system, became bored really fast, not amused
  - husband has laptop at home, 2 button mouse, Windows 7
- positive reaction to test, hopeful for computerised testing; "good idea"; happy to be tests on a computer, open to most things nowadays being on a computer

Notes:

excellent performance on most tests (more difficulty with spatial/language tests - sentence production, square puzzles, pattern construction (not recall)
help menu fine, nothing comes to mind
most tests seen as easier than pen+paper version, more intuitive, no differences expected at home, makes more sense to be done on a computer
overall difficulty with instruction screens; not clear what to expect without practice version or demonstration (audio/visual tutorial?) - felt like "diving in blind" to many of the tests, despite performing well after completing trial / practice items - instructions clear then
drag/drop vs. point/click - no preference "whatever works"
willing to take test again - est. 1/2 hour to an hour
tests were fun
would be likely distracted at home, mind would wander (FTD symptom), CG would need to be in room to keep on task; motivation using a timer/clock image and/or progress bar

test fixes/suggestions:

Temporal Orientation: "I don't know / guess" button? - unable to proceed without guessing
Symbol Digit Matching: number ~ digit terminology - wording consistency
- "click to continue" vs "click for next trial" unambiguous wording to facilitate the option of taking a break
Similarities : "e.g." / example given in instruction slide
Pattern Construction: 2 colours misleading - more than 2 colours? different for each trial shape?
Sentence Comprehension: automatic correct screen rather than "continue/done" button
- "start w/ smallest" instruction is a memory aid? remove? alter?
Arithmetic: "Arithmetics" is a typo? no"s"?
Sentence Production: make target drop zone for sentences more obvious
Square Puzzles: need "start over" option to work
Misplaced Object Recall: instruction unclear - attempted to drag misplaced item to corresponding image, rather than selecting the scene (step 1) and selecting the location (step 2)

the entire battery: clear "click to proceed" / "click to go back" buttons on all slides where this is missing / unclear, especially in Picture-Word-Pairs - currently many clicks are unprompted