Ducky and Sherman's CSCW Project 3

Intro

Project setting

What we will call the "Squares" project involves doctors in a particular, narrow subspecialty who are located at different locations across Canada. There are very few doctors in this subspecialty -- on the order of ten in all of Canada. This means that doctors in this subspecialty worked in large part "in a vaccuum", ignorant of the history of similar cases that had been treated elsewhere.

Doctors really only communicated with their subspecialty peers around twice per year, at large conferences in their larger specialty (not subspeciality) area. If they wanted to discuss cases, they would bring a bag of X-rays, patient files, and laptops, and grab a corner at the conference to discuss the cases. Usually, however, this was junior doctors seeking advice from senior doctors. In general, senior doctors did not consult with each other.

In 2003, a group of doctors started meeting regularly via computer conference several times per year in order to discuss interesting -- usually difficult -- cases. There are fewer than ten doctors who participate, and all are associated with teaching hospitals in major metropolitan areas. (There are only enough patients in this subspecialty every year to support a handful of doctors.)

In addition to the doctors, a number of other medical professionals are allowed to listen in on these case discussions: fellows, residents, clerks, and researchers. Usually, they play only a passive role in the actual conference.

One explicit goal of Squares is to come to cross-Canada consensus on treatment protocols, i.e. "if symptom X arrises, administer Y treatment". The field is evolving rapidly, and not all the doctors will be able to experiment with all of the new treatments; a goal is to share what each person learns with all the others.

Another goal is to identify which areas warrant more study. If they find that collectively, they are seeing a lot of patients with a particular project, then perhaps it's worth someone doing a detailed literature search.

Software

The conferencing software, which we will call "LectureNotes", that they use was developed by a medical devices company, and originally was intended for use training salespeople in a traditional "lecture" format: one person doing most of the talking and the rest of the people mostly listening.

The medical devices company provides LectureNotes as a promotional benefit to the doctors. The software is entirely Web-based, but requires a technical administrator at the devices company. The administrator deals with passwords, with setting up the conference, and troubleshooting technical issues. The administrator might also play an active role in turning microphones off and on during the conference.

@@@ picture of screen -- does Sherman have a better one?

LectureNotes takes up the full screen, and has the following sections:

• video: there is a small area at the upper left of the screen which projects the video of the person who is speaking or who last spoke. The video feed comes from a webcam on or near the speaker's computer, pointed at whoever is sitting at the computer. (This means that those without
• participants: a scrolling list of people who are logged in
• presenters: a scrolling list of people with special "presenter" status. Presenters have the power to manipulate or modify various things, while non-presenters can only observe/listen. In the way that Squares currently runs, all participants are also presenters, so this pane is redundant.
• agenda: not used
• slides: a large area (taking the majority of the screen) which is used to present PowerPoint slides. Those usually consist of pictures (X-rays, CT scans, MRIs) of the patients.
• toolbar: controls for many things, including "raising a hand" (i.e. requesting permission to speak), various ways of annotating the slides, advancing to the next slide or retreating to the previous one.

In the list of participants, there is an icon which represents a "hand up" that can appear next to a participant's name. Associated with that icon is a color: Red means that the person wants to speak, yellow means that they have been given permission to speak, and green means that they are speaking.

The presenters can "draw on" the slides, and have several different drawing tools to select from.

To speak, a participant holds down the Control key. That puts the "hand up" icon next to their name. We were unclear about exactly how people were cleared to speak: one person told us that they were always cleared to speak, while another said that the medical devices company's administrator pushed buttons to enable/disable microphones.

Data collection methods

We did one observation of the Squares conference, then did two follow-up interviews. The first follow-up interview was with two researchers, P1 and P2. The second interview was with a senior doctor, P3.

Observation

We observed a group at a local hospital the day of a Squares conference. We arrived a bit early, and observed during the setup, the Squares conference, and then for a little bit afterwards. Kelly Booth advised us not to tape the conference.

Ahead of time, we had agreed that we would divide up duties as follows:

Ducky	Sherman
draw diagram of room	draw diagram of room
watch the screen and who can see what	take (very occasional) pictures
keep basic time-annotated transcript	watch for people referring to visual artifacts

During the Squares conference, doctors P3 and P4 each sat at a computer at their desk in their respective offices. P1 and P2 sat and watched in P3's office. P1 and P2 had told us that they expected more people, but aside from a staff surgeon who entered, stayed for about ten minutes, and left, there was nobody else in the room.

Three remote doctors participated (R1 through R3).

Because we had been led to believe that there would be a lot of people in the room, we stuck ourselves to the back wall of P3's office. Partway through, when it became apparent that nobody else was coming and that our position didn't let us see what was on the screen, Ducky moved up. She lodged herself just behind P3 and as close to the left wall as possible to minimize her appearance on P3's webcam.

Afterwards, we chatted with P1 and P2 briefly.

Follow-up interviews

P1 and P2

After the Squares, we arranged individually with P1 and P2 to interview them. Kelly Booth had told us not to tape Squares, and we had misinterpreted that to mean not taping the interviews, either. Before the interview, we agreed that Sherman would focus on taking notes and Ducky would focus on interviewing.

The best-laid plans of mice and man often go astray, and so did the plans of Duck and Sherman.

• Even though we had scheduled interviews with P1 and P2 at different times, they assumed that we would interview the two of them together. They wanted to be interviewed together. As they were doing us a favor, we couldn't really force them apart.
• While we know that we are supposed to interview people in their offices, P1 and P2 both resisted, and for good reason. Both offices had other peoples' desks in them and precious little space to sit. To interview P1 and P2 in either of their offices would have been very disruptive to their officemates. Thus we went down the hall to a conference room.
• We didn't have a chance to discuss how we would deal with a two-person interview, and Sherman did not write down who said what.

P3

Doctors are famously busy creatures, so we were quite pleased when we got 15 minutes on P3's schedule at 0730 one morning.

We arrived at the hospital at 0715 (with a voice recorder this time!), and stopped at the hospital coffee shop to do last-minute coordination over coffee, only to discover P3 at the coffee shop. He was ready to go, and seeing the chance to get more time with him, we jumped at the opportunity.

That morning, we discovered that the voice recorder's batteries were dead. He was able to recharge them in the car; we realized that this was sub-optimal, but figured they would last the 15 minutes that we had. (We couldn't just stop at a convenience store and buy new batteries, as this particular recorder had a built-in battery.)

It turned out that P3 was very unhurried that morning. Apparently, he needed to be at the hospital early in case some problem cropped up in pre-op, but he didn't have any specific obligations. He generously gave us about an hour of his time, during which time our recorder died, was recharged, and died again. Hence our notes from that interview are partially a verbatim transcript and partially just recopied handwritten notes.

P4

We contacted P4 to request a follow-up interview, but he did not respond.

Overview of data

For the had the following focal points:

• How does the speaker indicate to others what visual artifact he/she is referring to?
• How does the physical layout affect the collaboration?
• How do they manage screen real estate?
• How do the participants manage turn-taking?
• How do status levels affect the collaboration?

Methods of analysis

@@@ uh... what is this? Basically, we just looked at the transcripts, thought about it, and discussed it.

Findings

Turn-taking

Turn-taking worked well: P3 continuously scanned the list of participants to see who had their hand up, and called on people explicitly. (Unfortunately, the list of participants was longer than the participant window. P3 had to scroll through the list of participants essentially continuously in order to see who had a hand up)

We witnessed only three interruptions -- places where one person spoke over another, and it was never disruptive. In one case, it was a very quick answer to a question. In another, someone who had had the floor needed to make a quick amendment. In the third, @@@?

However, we were told that this had not always been the case -- that the participants had had to learn how to interact.

P4: "A new person can ruin a session by holding the control key down." (i.e. keeping the mic open)

P4 complained about how anybody could "move the slides around" [presumably meaning advance/go backwards].

P4: "They think it's a dialog, but it's not."

One aspect of the moderated approach is that people would sometimes have to "save up" their comments until they had the floor. A "hand up" signal doesn't convey urgency -- it's not clear if the participant wants to comment on something that was just said or about some new topic. P4 mentioned that sometimes by the time your turn came, what you had wanted to comment about was long gone.

Physical layout

We observed that P3 appeared to be really uncomfortable sitting at his computer, craning his neck backwards. It looked to us as if he was wearing bifocals and needed to tilt his head back in order to read his screen. Several times, we witnessed him wiggling his shoulders as if to get tension out.

Because the webcam and microphone were both pointed at the person sitting at the computer, other people in the room were essentially shut out of the discussion. P3 later commented that he wasn't sure there was a active role for the junior people in Squares: that the role of the learner in that situation was to learn, not to advise.

We did observe P2 ask a question during Squares.

Side channels

At one point during Squares, P3 rolled his chair back to where Ducky was and made an enthusiastic comment. It appeared that he was excited about how well the process was working, and needed to tell someone. In a face-to-face meeting, like a trade show presentation, he probably would have leaned over to his neighbor and enthused. Because there were no side channels to peers that he could use during Squares, he enthused to Ducky instead.

We also observed P1 and P2 whispering to each other during Squares, even moving their chairs together to be able to better communicate. When asked about that, P1 and P2 said that they were talking about what was happening, especially regarding a case that had a big surprise in it.

Emotional support

We noted that the gathering was very comfortable and collegial. Several of the doctors made jokes, including light-hearted teasing of each other.

Based on our interview with P3, we think that a significant benefit of Squares to the doctors was emotional support. The type of surgery that they do is extremely long and gruelling, all patients respond differently, and there are frequent trade-offs that the doctors are forced to make. In particular, decisions about how aggressively to treat something are difficult: what do you do when a more aggressive treatment is more likely yield a higher chance of living but also a lower quality of life?

Referring to artifacts

The software gave the ability to draw on the screen with several different tools. During Squares, we saw two different doctors pointing out something on the screen with those tools. We also witnessed one incidence of P3 referring to something on the screen and circling his cursor (which we believe nobody else saw) around an area.

Setup

The amount of work that happened before the conference starts was significant. The doctors take turns preparing cases and moderating. Each session has a theme, and so three to five cases need to be found on that theme.

When Vancouver hosts, P2 does most of the legwork. P2 says that P3 and P4 will give her a starting point that might be as sketchy as a name, or a description of a case and a rough timeframe. P2 then has to track down the charts, find the appropriate films, and sit down with a radiologist to figure out which films are the most illustrative and hence the best use. For older, non-digitized films, she needs to digitize them. She then pulls all the images together and creates a PowerPoint presentation, and sends that to the administrator at the medical devices company.

When other sites host, P2's job is easier. The remote site sends her either a polished PowerPoint presentation or simply raw images. She does tweaking as required, and forwards it on to the service administrator.

On the day of the conference, it takes a while for everybody to get set up. Every hospital has a different IS system, and sometimes, the IS department has to do something extra to get the software access through the firewall. One time the URL that was mailed out was incorrect, and one of the doctors didn't know how to go to the main login page and navigate from there. Sometimes passwords are forgotten.

Setup problems are exacerbated by there being as much as four months between sessions to forget how things were done last time.

P2 does the setup basically on a volunteer basis. She is part-time and not paid for Squares administration work. P3 has said that he couldn't do it without her. Thus Squares depends upon what is essentially a charitable contribution.

Video

People were uncomfortable in small ways with the camera.

P2 didn't like how the camera portrayed people:

P2 (to Sherman and I during setup): "Oh, the camera is looking down. I don't like how it's looking down."

There were several jokes about the cameras. During the setup phase, P4 said to us:

P4: "I'm not as good-looking [as P3] so they won't let me have a camera."

Later, during the conference, P3 made a comment about the video:

P3 (to R1): "I'm not as good-looking as you, but we'll have to live with that."

R? [presumably R1]: "Well, as long as you understand that."

It wasn't clear how the source feed for the webcam was selected/switched. Different participants had differing beliefs about that.

It wasn't clear to us how useful the video was -- it was relatively low quality, the doctors didn't look at the camera (so the eye contact was "wrong"), and it constrained the doctors' movement. (If they leaned forward, it would see the top of their head; if they leaned back, it would see their torso.) However, P3 was adamant that the video was useful. He felt that anything that added richness to the interaction was good.

If one of the major benefits is emotional support, than it might well be that video is important. A picture of someone might have a much stronger emotional resonance than simple audio.

Audio

The sound quality of the remote voices was spotty. Usually it was intelligible, but it varied.

P3 explained that every site had slightly different hardware, camera, microphone, and bandwidth. Thus quality was uneven over all the sites.

Location / participation

P4 prefered to work in his own office instead of coming down to P3's office where everyone else was.

P4: Sometimes when it's slow, I can get other work done.

Improvements desired

P3 expressed a number of concrete and abstract desires for improving the Squares experience.

Many of his frustrations centered around the hardware. He wanted a bigger screen and a wide-screen format screen. He wanted everybody to have the same (presumably high-quality) hardware/microphones/cameras. Part of the constraint was financial, but part was informational: he wanted someone to give a recommendation on which cameras/monitors/CPUs/microphones he should buy.

He felt that there were some changes to the system that would be helpful:

P3: I think it could be a little smoother to bring people in and out, dampen the mic on the non-speaker, and there is clumsiness in hooking up to various stations.

He also was very frustrated by the limitations of PowerPoint, but was unable to articulate precisely what the problem was or how it should be fixed. He expressed the opinion that that was the job of people with good communication skills to figure out.

---

It was interesting to me how some good material surfaced during times that were not formal interviews or observations, just smalltalk.

• Set ALLOWTOPICCHANGE = DuckySherwood, ShermanLai