I think this is useful info. There's also some previous user studies you've done lying around somewhere else too, yes?
I think it's unlikely for anything to come out of any of it without trac tickets, though. I've already forgotten where you even posted the previous user studies.
Thus spake Andrew Lewman (andrew@torproject.is):
I spent the past week in Sweden, attending the Stockholm Internet Forum, http://www.stockholminternetforum.se/, for part of it. I made a number of tails 0.10.2 usb sticks for people on request. I also asked a lot of people their impressions of Tor and Tails. I received a plethora of feedback. All 8 people are involved in the Internet Freedom policy, technology, or freedom of speech communities. They had very different levels of self-assessed technical skill. The 8 people represented 6 countries.
tldr; 8 people were tested, 8 people had trouble doing simple things with tor browser in Tails. Issues 4-6 are directly related to Tor.
And by asked, I mean, I stuck them in front of my laptop, put the usb stick in the computer, and asked them to browse to their favorite site. No one wanted to be video recorded, even if I offered to only record the screen and not audio.
Everyone managed to power on the laptop and wait for Tails to boot.
# First issue: Language selection
The first issue was on the language selection screen. 4 of 8 people were confused why it was called "Debian Live System" and not "Tails Live System". 8 of 8 knew what language selection meant, but weren't sure how this mapped to Tails.
# Second issue: wifi and tor browser
The tor browser starts up before the wireless is configured. The tor browser then reports a proxy error. With some prompting, all 8 figured out the wifi and then didn't know what to do. Tor does keep trying to load, and takes forever because it needs to download the entire directory. Users have no feedback as to what's going on behind the scenes because vidalia is hidden.
8 of 8 waited patiently for something to happen on the screen.
# Third issue: green onion
3 of 8 people saw the green onion appear in the menu bar up top. These three people hovered over it and saw the 'Connected to the Tor Network' message. No one knew to double-click on it to get a menu of other things to do. No one knew to right-click on it to get the drop-down menu. They were presented with the default check.torproject.org 'congratulations' page and then sat there.
# Fourth issue: check.tpo is not helpful
8 of 8 people saw the default check.torproject.org site telling them 'congratulations. Your browser is configured to use tor.' 7 of 8 people asked 'where is my browser?' The one who didn't ask this question was already a firefox user and recognized the interface. 0 of 8 understood what the IP address message meant. Comments ranged from 'is that different than my current IP address?' to 'what's an ip address?'
As an aside, when showing someone TBB on their own laptop, they saw the check.tpo site, and then went to Safari and started it up. When asked why they did this, the answer was 'safari is my browser. this says your browser is configured to use tor.'
No one used the language selections at the bottom of check.tpo, nor even understood why they were there.
# Fifth issue: exit relay congestion/failures
8 of 8 people tried to get to their own sites. 'I wonder what my site looks like when I'm anonymous' was the most common comment (5 of 8). For 6 of 8 people, their site didn't load at all, and tor browser reported their site was unreachable. All 6 then tried to go to google search in their own language; meaning google.es, google.se, etc. For 3 of those 6, this didn't work either. They gave up and assumed tor was broken or was censoring their destinations.
I intervened, opened the vidalia network map, closed the circuit in question, and asked them to repeat their browsing.
5 of 6 were able to get to their sites now. The one that was not able to had the same exit relay as last time, Amunet1, in a new circuit and just couldn't get anywhere through it. After yet another new circuit, they could get through to everything.
The user has no feedback as to why their site didn't work. And tor assumes everything is working fine.
When asked "please find a video you like", they all went to youtube. Most of the videos they wanted to see resulted in 'This video is currently unavailable.' 8 of 8 assumed it was because youtube was blocking tor, not because the video is flash-required. 2 of 8 started randomly clicking videos suggested by youtube to see if any of them worked. Eventually, 2 of 8 got videos to work with youtube and were amazed it worked at all.
# Sixth issue: no flash, no warning
2 of 8 people had flash apps on their website. 4 of 8 had ad banners that used flash. All were surprised at the red outline with a snake in it appearing instead of their flash apps. None understood what happened.
After an explanation, one person suggested changing the red outline with snake to an actual message written inside, along the lines of 'this app blocked for your protection. click here to unblock it.' I explained why that wouldn't work (because there is no flash, java, silverlight plugins installed) and their answer was 'then do not show it at all'. Inside noscript, I unchecked the 'show placeholder..' option and had them browse again. they were happy. It seems if the user cannot do anything about the blocked apps, not showing them may be preferred.
# Seventh issue: shutdown
I asked all 8 to shutdown tails and let me know when they thought their data was safely no longer on the system. 1 of 8 figured out how to shutdown tails by clicking the big red button in the upper right corner. The rest hit the power button on the laptop.
After rebooting, i showed them all they could just pull the usb drive to do it as well. As soon as tails started shutting down, they all assumed everything was safe and tried to power off the laptop.
-- Andrew http://tpo.is/contact pgp 0x6B4D6475 _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev