Controversies in our field

[avrambaskin at comcast.net]

There is a difference between observing users in their natural environment and fostering naturalistic behavior when participants perform tasks.  By naturalistic, I mean as natural as possible within the restrictions that Mary notes as being inherent to laboratory studies.  So, for example, the participant performs the tasks as "straight through" as possible, with the only interuptions being to remind them to "think out loud" when necessary and providing minimilist responses to participant questions -- for example, "I can answer your questions at the end of the session".  The participant is still thinking out loud during the session.  The only retrospection is the participant's responses to my questions.

I have never participated in a study in which there was noone in the room with the participant.  I have found that when the facilitators position is offset to that of the participant -- that is, the facilitator is behind and to the right or left of the participant, the participant quickly forgets that the facilitator is there.  Just thinking out loud (pardon the expression) it seems to me that having a person in the room may give the participant a measure of comfort that they would not have if the facilitator is not present.  If, all of a sudden the voice of big brother comes on over the intercom to remind the participant to think out loud, I would think that could be a bit unnerving.

On a somewhat related note, how many people use time on task as one of the metrics in usability studies?  When I was in graduate school, everyone used this, mostly, I think, because it was a metric that was easily derived from video recordings or with a stop watch.  I think the variability introduced by thinking out loud (not everyone thinks out loud at the same rate) makes time on task an innacurate metric.  And probing the participants during the session exacerbates the variability. 

-------------- Original message -------------- 
From: "Mary Deaton" <mmdeaton at gmail.com> 
In a user observation test in which a user is completing tasks, I never sit in the same room. I do however, communicate with them in a limited way, such as prompting them to speak aloud, or asking them to expound on something they said or did. This interaction is limited, but I am not taking copious notes as I watch because I am recording the visual and audio using Morea and putting index points into the record. This allows me to take more meaningful notes,  rather than trying to capture their talk verbatim. I will also probe on issues I have noted once the task work is done. If we really want users in a natural environment, we ought to be going into the field to observe. The lab is, by its nature, unnatural, and wherever we, the researcher sits, it will never remove whatever impact the lab has on the user from our results. 

A group at the UW did a study of the validity of doing retroactive think-aloud in which they used eye-tracking.  This might shed some light on the general topic of where does the researcher sit.

Guan, Zhiwei, Shirley Lee, Elisabeth Cuddihy, Judith Ramey,  "The Validity of the Stimulated Retrospective Think-Aloud  Method as Measured by Eye Tracking,"  technical paper, CHI '06 (the annual conference of the ACM Special Interest Group on Computer-Human Interaction), Montreal, Canada, April 2006.


On 9/3/07, Avram Baskin < avrambaskin at comcast.net > wrote:
1.)     Should facilitators/moderators sit next to the user, carrying on a dialogue with them, and taking only limited notes during the study (a.k.a. "in front of the glass"), or should they sit in another room or apart from the user, with limited interaction with the user, taking copious notes ala a court stenographer (a.k.a. "behind the glass")
This issue isn't necessarily binary.  I still sit in the room with the participant, but I have gradually stopped probing participants during usability tests in the interest of making the experience as naturalistic as possible.  I write down my questions and rely on their retrospective responses and showing them screens as needed.

Chauncey said 



-------------------------------------------------------------------------------------------------------------------------------
This is a complex issue.  The answer partly depends on the state of the software and the goals of the study and your resources. If you are interested in summative testing (measuring against specific quantitative goals, then you probably don't want anyone in the room).  So, this is an "it depends" answer, though I could probably put together a matrix that would be a robust guide to when you should have a moderator.  Also, are you talking about a single moderator for the thinking-aloud usability test or are you talking about observers, including a moderator (which is OK for participatory design, but generally not for "testing").  If you are alone (and without a screen recording system with audio) then you are notetaker, moderator, host, and trouble shooter.  

Notetaking is a really hard thing to do and one controversial issue is whether developers, writers, product managers, etc. should be the ones to take notes for analysis (and not necessarily a usability notetaker).  I think that taking notes in any complex environment is very hard and involves watching and listening.  If you take the notes from any group of people watching the same tape or session, the notes and problems are often quite different.  I think that note-taking is hard because it is not always clear when a "problem" begins and ends in complex systems.  I would prefer a very trained notetaker and would accept other notes as well, but would not put all the burden of notetaking on people whose job is not to observe users.






--- You are currently subscribed to stcusesig_l as: avrambaskin at comcast.net To unsubscribe send a blank email to leave-stcusesig_l-345582G at lists.stc.org 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.stc.org/pipermail/stcusesig_l/attachments/20070904/eed2cffe/attachment.html