C-TOC - Cross-Cultural Review Panel
Notes by Matthew Brehmer, M.Sc Researcher, UBC Dept. of Computer Sc.
First Iteration: May 27, 2010
Introduction
- Setup of C-TOC prototype on Douglas College computer lab: no security/admin issues - macros working successfully
- refer to literacy education for seniors / immigrants - largely computer-based
- refer to UBC learning exchange w/ DT east side (apparently 40yo = senior)
- Claudia's introductory presentation
- cultural advisory coordinators: Kymberley Bontinen, Patricia Juvik, Sai Roshni (Priya) Raju; CHCP (centre for health and community partnerships) project manager: Marina Niks
- cultural advisory panel members (5, one missing)
- Chinese, Japanese, South Asian, Southeast Asian, Latin American communities represented
- Vivian Lam, Sayuri Sugawara, Kamaljit Kaur, Lam Dang, Karla Maranhao
- recruited from community groups that interact w/ seniors in their respective communities
- representative of health-care / extended care related positions, nursing, counselling
- all members of the panel are immigrants
Interactive Prototype Session Notes
- users: 5 panel members, M. Niks, R. Hsiung, S. Raju, K. Bontinen, P. Juvik (10 total)
- difficulty realising what is a mockup and what is interactive (i.e. drop-down menus in introductory slides) - distinction may not be clear;
- suggestion: use a warning label in the future?
- red prompts for referring to questionnaire not obvious enough - centre on screen rather than top-left?
- suggestion: include pictures representative of each test on the questionnaire as reminders / in case they have skipped ahead and forgotten the questionnaire
- drag-and-drop habit hard to shake - everyone had difficulty with click-and-release moving of objects - some skipped the intro slide "move the blue circle into the square"; difficulty arose in sentence comprehension, which depended on this interaction; by this point many had developed an understanding of this interaction
- will older users be similarly biased / used to traditional drag-and-drop? it requires constant motor pressure - did the panel understand this rationale for using click-and-release?
- suggestion: try out traditional drag-and-drop in next prototype version and ask what is preferred
- pattern construction instructions not well understood by some users; dragging to other areas of the screen aside from the target zone; for instance, covering the source shape;
- one user puts all the component shapes back into the source area after completing the target shape; not sure why?
- clicking on trails test not required but done by most users
- suggestion: state that it is not necessary, provide animated instruction sequence
- click+drag on a shape misaligns the cursor, decoupling it from the shape causing confusion and the possibility for the shape to be dropped off-screen, or being unable to click-to-release shape unless the cursor is clicked on top of the shape;
- not much to be done - a fault of the PPT macro
- square puzzles not fully interactive - some lines are still missing the drag/drop macro (#7 in particular)
- "stack on top" instruction in sentence comprehension has potential for multiple interpretations: some users layer shapes, some place shapes above one another on screen
- some go/no-go slides not advancing / hyperlink is broken and leads to an erroneous place in the slide deck
- sentence production instructions are still missing the additional instruction to "use as many words as possible"
- computers low on memory after opening 16 slide decks (will be resolved in non-ppt version)
- most participants finished session in 1h20min; final participant in 1h30min ("too long!")
Focus Group Notes
- Could handwriting be used as a diagnostic tool?
- general consensus: this test will work well for well-educated, high-income, well-integrated people
- problem w/ mouse drag-and-drop were vocalised
- suggestion: avoid this by using touch-screens
- request for audio instructions as optional feature for all test instructions; other instruction formats could include a flow diagram / animated step-by-step instructions
- more practice sessions requested
- language throughout the tests needs to be overhauled, cultural-specific terms discarded; possible to translate into other languages?
- Similarities test especially sensitive to language norms
- cultural-specific names in item recognition test misleading (i.e. "stationery item")
- suggestion: allow family members to translate, but not help with task
- provide an "I don't understand" option, a flag to de-validate a single test if necessary
- community centre use case scenario: group members help one another take the test rather than family/caregivers (peer support)
- identified shame in cognitive illness, acknowledging it in the household, esp. in South Asian community
- suggestion: provide feedback on each test: descriptive time elapsed, # items correct, where you scored comparatively to pop. norms, past times taken
- how to deliver potentially bad news over the net? instead direct to community resources regardless of test performance
- next panel meeting (late October 15/28/29?)
- suggestion: provide a shorter evaluation version of the test for cultural advisory panel (i.e. less trials per test), provide web-accessible version of next prototype version in October, allow them to take + fill out survey from home - likely not in PPT, but in interactive programmed version (Flex?)
- longer focus group next time, allow discussion on interactivity of each test
- more focus group members to be recruited
Second Iteration: October, 2010
Introduction
- Setup of C-TOC prototype on Douglas College computer lab: no security/admin issues - macros working successfully
- refer to literacy education for seniors / immigrants - largely computer-based
- Claudia's introductory presentation
- cultural advisory coordinators: Kymberley Bontinen, Patricia Juvik, Sai Roshni (Priya) Raju; CHCP (centre for health and community partnerships) project manager: Marina Niks
- cultural advisory panel members (5, one missing)
- Chinese, Japanese, South Asian, Southeast Asian, Latin American communities represented
- Vivian Lam, Sayuri Sugawara, Kamaljit Kaur, Norma Sanchez, Gita Rafiee
- recruited from community groups that interact w/ seniors in their respective communities
- representative of health-care / extended care related positions, nursing, counselling
Interactive Prototype Session Notes
- users: 6 panel members, M. Niks, S. Raju, K. Bontinen, P. Juvik (10 total)
- observers: MB, HL, CJ, CT
- session started ~40 min late, finished ~45 min behind schedule (longest participant session 1h15min, shortest around 45min)
- intro/help menu/throughout CTOC
- tooltips on buttons and links may be confusing
- help menu has no exit button (must click on 'help' again to exit
- by moving "back" from slide 2 to intro screen, you are unable to move forwards again;
- too much text in help dialogs
- pull-down menus not yet interactive
- red questionnaire response cues are being missed - not obvious enough
- "Click" here to proceed to next screen on page 1 - ambiguous as to where to click
- yellow dot to click on unresponsive for some users
- "are you still there?" prompt premature, timeout too soon
- PWP - after clicking on stimulus once, there is a long pause before feedback is given - potential source of spurious clicking or confusion
- inform user of pause? mouse-over the name - only require 1 click?
- on "correct" screen - show PWP before next trial?
- SDM - stopwatch image confusing - small clicking target - long pauses here
- SC - inconsistency in feedback given after practice problems: "correct" vs. "good"
- "overlapping" vs. "on top"
- obvious drag/drop issue
- SIM - no pictures for "fame" and "money" stimuli
- WR - no dedicated questionnaire page - tied to PWP
- MOS - need to click twice on misplaced object
- would "click on object that doesn't belong" work?
- contrast problem - potential need for hiring a graphic designer to design scenes and stimuli
- bigger objects required
- TRA - automatic generation of lines on mouse-over confusing and likely invalid - people will click anyways; * numbers and letters culturally dependent
- SqP - one trial has impossible instruction
- obviously difficult (for healthy adults)
- PC - wording instructions too small
- reconstruction: "... like these" : instead of "."
- SP - dra/drop macro missing for some stimuli
- GNG - instructions too lengthy, not being re-read while they are subtly different
Focus Group Notes
- demographics: acceptability of CTOC in 80+ - computer literacy issues
- children will bring CTOC home to parents - dependent on culture and living situations - others will prefer company of peers in public spaces - knowledgeable peers or nurses
- high trust and confidentiality among peers in some communities, others among family
- knowledge of English still an issue; more graphical instructions needed to address language issue;
- provide a limited version w/ no language used, full version w/ language?
- high level of education, computer literacy, cultural knowledge required - can cultural content be more general?
- test could agitate depression or severe anxiety; can people break at will? a clear way to pause the test needed; test takes a lot of energy, very fatiguing
- consider a practice version of the test to be taken before undertaking the full version?
- adjusting test difficulty dynamically dependent on early performance - increasing difficulty may conflict w/ fatigue - confounding variables
- "sorry try again" discontinuation rules after consecutive failures
- multiple clicks and pauses a hassle, spurious clicking - must strike a balance b/w always needing to click and not clicking
- moving objects very difficult
- variable scoring based on location of test-taking and current anxiety score?
- practice problems / instructions:
- buttons top small throughout
- use audio for instructions and feedback but not for test content - may be a distraction - optional audio
- graphical sequence requested - demonstrating low level mouse clicks - accessible via help menu
- graphical instructions well received, increased confidence and reduced stress - clear pictures
- could instructions affect test validity? biases?
- provide multiple options for different kinds of learners
- itemise instructions to encourage re-reading of instructions with subtle differences (i.e. Go/No-Go)
- interruptions / distractions
- physical noise - preventative, doctor's office scenario? use of sound-cancelling headphones? could still need to shut out loud conversation
- resumption cues: could provide an automated summary, recap illustrations, one-step back
- strategy: resume scoring on next trial - several cutoffs for "are you still there?" several restart points dependent on length of interruption
- bathroom breaks should be considered
Third Iteration: February, 2011
Introduction
- Setup of C-TOC prototype on Douglas College computer lab: no security/admin issues - macros working successfully
- UBC attendees: MB, RH, HL, CL, WW (CJ ill)
- Robins's introductory presentation
- cultural advisory coordinators: Kymberley Bontinen, Patricia Juvik, Sai Roshni (Priya) Raju, Marjory Ditmars; CHCP (centre for health and community partnerships) project manager: Marina Niks
- cultural advisory panel members (4, one missing)
- South Asian, Southeast Asian, Persian, Latin American communities represented
- Kamaljit Kaur, Norma Sanchez, Gita Rafiee, Fariyal Dhirani
- recruited from community groups that interact w/ seniors in their respective communities
- representative of health-care / extended care related positions, nursing, counselling
- next panel meeting likely in May-June
Interactive Prototype Session Notes
- users: 4 panel members, CHCP coordinators (6 total)
- observers: MB, HL, CL, RH, WW
- longest participant session around 75 min, shortest around 60 min
- delayed start (surveys, headphones needed to be distributed)
- General / Navigation
- button affordances are required throughout (mouseOver, mouseDown, disabled feedback, an audible click on mouseDown);
- ppt hyperlink mouseOver modal frames are shown currently (confusing, meaningless hyperlink text is shown)
- consistent button UI needed throughout: colour, text colour, gradients, border, font, font size, bevel, etc.
- many nav. buttons have poor-contrast: black text on blue background.
- help buttons are currently different from nav. buttons (colour, shape);
- I suggest a permanent control bar with small set of icons for help and navigation, visible throughout the test; pause button, progress bar visible between tests; consistent button UI and affordances, redundant coding of button meaning (text and icons); selected help menu items could be enabled/disabled depending on where the user is in CTOC (between tests, during tests, etc.)
- click to pick up, drag, and click to drop instructions still insufficient; default drag-drop behaviour overrides; this will be eliminated in programmed version;
- computer-literate-biased terminology used throughout: "Back", "Icon", potentially more?
- "Click "Next" to continue" should be added to all introductory test screens (currently just the title of the test and the "Next" button are shown); "Next" button could be enlarged;
- Introduction
- drop-down menus still not implemented (explained by small note - it is difficult to differentiate between limited interactivity notes and genuine test materials);
- shortest completion time of the introduction was about 12 min
- Help Menu
- What does this test mean option elicits a "available with every test" dialog; this doesn't provide any context: it is two clicks away from the test, so context is difficult to recall without a visual reminder and/or additional text; consider embedding a screenshot of the current test screen in the help UI, to provide context;
- Each option in help opens a modal window, which is unnecessary (requires a button click to close / return to the main help menu; the main help menu options should always be visible when the help menu is shown, as should the progress bar - these components could be grouped on the left of the screen; the right of the screen could be populated with help content, whenever an option on the left is selected;
- In some help dialogs, the user is shown a screen of text, at most 7 lines of text in a single paragraph; consider breaking this up into smaller paragraphs with headings or re-arranged into a FAQ format, include visuals;
- There is no indication of where "Back" takes you, nor what "Back" does; we cannot assume familiarity with the meaning of "Back" as used in web browsers or file tree navigation;
- Animated transitions (revealing or closing modal windows) may be confusing for some users; eliminating the need for modal windows eliminates the need for distracting animations;
- If a help menu option is available and selected during a test, there should be some indication given that the current test is paused;
- all help menu options should not be available during time-sensitive components of trials (i.e. available during the encoding stage of a sentence comprehension trial, but not during the action phase);
- progress bar needs to be more visible; panelists remarked that they would like to be shown progress during CTOC, perhaps between tests, without having to look for the progress bar; include on conclusion screen for each test? the progress bar must not be confused with performance;
- Picture-Word Pairs
- "Click "Next" to continue" should be added to "Sorry, Try Again" screens;
- 4 lines of text on practice problem
- Temporal Orientation
- large block of text is given on demo screen: counted 6 lines;
- button affordances needed for calendar grid buttons;
- Symbol-Digit Matching
- button affordances needed to differentiate between icons and buttons;
- Similarities
- 4 lines of text are given for practice problem; after practice problem, 5 lines of text are shown on a screen with no additional interaction required;
- some trials without images - is this deliberate?
- panelists found this very confusing, largely a language issue and familiarity with abstract/concrete representations in English
- Word Recognition
- redundant navigation cues needed;
- should the category word get more emphasis?
- only the first 2 puzzles have images - is this deliberate?
- Pattern Construction
- it is not clear that any interaction is possible or required on the example screen; this example interaction should be forced - the "Next" button should only be enabled after the correct interaction is completed;
- 6 lines of text on example screen;
- shapes can overlap bounding boxes; this problem will be eliminated when implemented programmatically;
- Sentence Comprehension
- it is not clear that any interaction is possible or required on the example screens; this example interaction should be forced - the "Next" button should only be enabled after the correct interaction is completed;
- 7 lines of text on first example screen (no interaction is required);
- add instruction: "Read instructions carefully because you cannot go back"
- very long instructions for some trials (no suggestions, this is the nature of the test)
- Trails
- the click instruction is misinterpreted: one subject repeatedly attempted to click on 2 before realizing she had to start at 1; the instructions should be supplemented with this instruction;
- an alternative approach could involve an elastic string metaphor: clicking on 1 could create a direct line segment between the 1 icon and the cursor, clicking on the next icon (A) in the sequence anchors the line segment at that icon, with another line segment generated from (A) to the cursor; in other words, a line segment connected to the cursor is always shown until the puzzle is completed or dismissed;
- visual feedback that an icon has already been clicked should be given (i.e. border, text, or background colour change)
- practice #2 has no further instruction other than "Practice";
- are these trials not timed? there is no icon illustrating this; is this deliberate?
- Arithmetic
- number grid button affordances needed;
- which part of the start timer icon is a button? is it all a button? if so, better affordances needed;
- Pattern Reconstruction
- this version contains more puzzles than were originally in the PC encoding puzzle set (likely overlooked when new cultural advisory panel version was created)
- shapes can overlap bounding boxes; this problem will be eliminated when implemented programmatically;
- Misplaced Object Search
- button / background contrast needs improvement; full-screen images are unnecessary - the image could be placed in a frame, with a navigation bar along the screen edge;
- there is no instruction as to what to do once the misplaced item is enlarged; some feedback (i.e. "Click to continue") should be provided;
- Sentence Production
- 4-5 lines of explanatory text is given on each trial, with differing emphasis and size;
- some sentences require more vertical space, using 2 lines of text;
- when implemented programatically, word icons should snap to the sentence area, and allow for snapping between already-placed words; automatic re-arrangement of words when middle words added/removed;
- Panelists remarked that the interaction here in this test needs re-thinking
- Square Puzzles
- some explanation is needed to mention that lines cannot be rotated or twisted;
- additional instruction is needed w.r.t. incomplete squares/ extra lines
- each panel member experienced considerable difficulty with this task;
- ability to place lines above one another should be disabled - lines should snap back to original position on attempted drop;
- Go No-Go
- no interactivity on instruction screens; 4 lines of text on instruction screen; 5 lines of text on later instruction screen;
- Misplaced Object Recall
- button affordances and contrast with background image need improvement; scene selection buttons need affordances (border highlights?);
- Conclusion
- no interactivity on this screen;
Focus Group Notes
- acceptability
- CTOC developers are English-speaking computer-literate students, out-of-touch with immigrant/refugee communities;
- expected that only 10% of older refugee in Latin-American community could use the test, even if translated to Spanish;
- willingness to take the test is not a problem; usability and language problems will be the barriers to adoption;
- CTOC easier than MMSE, but assistance in administration still needed;
- self-administered version difficult to imagine;
- family, GP assistants or receptionist nurses, home nurses, research assistants, community facilitators would need to administer the test in its current form;
- 5 years from now, more older adults will be ready to take the test;
- context of use
- panelists to poll their communities wrt to computer use, context of computer use, types of computer programs used, frequency of computer use, willingness to take a computer test and what the desired context should be
- older adults who use computers tend to use games, Skype (not universally), Internet, email, Word;
- many older adults use cell phones, used primarily for voice conversation; many tend to avoid expensive communication (i.e. video chat)
- 50/50 split on opinion wrt to use of CTOC at home: family settings may be distracting or not private enough for some communities;
- also a risk associated with taking CTOC in a public setting;
- potential venues: assisted living centres, recreational centres with computers, family doctor's office, over the web at home; community centres, adult day program centres;
- home-care nurses with a laptop to administer test? time to administer test a barrier here;
- long-term care assessors to administer test?
- language
- some words a barrier, but panelists convinced that someone with a grade 7 education could take the test; variability between cultures still possible (apparently gr. 2 Iranian arithmetic skills = gr. 7 North American arithmetic skills)
- trails test involves concepts that could be untranslatable;
- general usability
- using the mouse, clicking, dragging difficult for older immigrants and refugees
- additional graphics, illustrations would be helpful throughout (esp. illustrations for Arithmetic test)
- help menu:
- not useful in its current form; too much text, text too technical; help was unavailable when actually needed (i.e. Trails test);
- "need to take a break" option in help menu not yet implemented;
- sub-test-specific help material still absent;
- "what does this test mean?" option needs to simplify and reduce the text, or eliminated altogether
- progress bar went missed by many of the panelists, but would be desired; improved visibility required;
- good consistency between sections wrt to navigation;
- too much clicking throughout; users will be fatigued;
- audio
- audio helps - audio should be either on (automatic) or off entirely, not toggled for each instruction (Manual audio); more control;
- audio could deliver incorrect response prompts, supplementing visual information, potentially with graphics or animations;
- CL's voice was found to be pleasant
- practice problems / instructions:
- more practice is needed for the tests; could it be more like a game? practice on the games could help performance on later tests; games would not score or have any diagnostic/screening repercussions, aside from practice performing actions;
- practice = readiness to take CTOC; must overcome lack of computer skills (potential for optional additional practice for those without computer experience);
- interruptions / distractions
- depends largely on context;