Last January I had an interesting discussion with Meryl Evans and Bill Moore on usability testing (direct link to part 1 and part 2). The gist of it is doing usability testing in a sandwich shop, and in the course of our discussion a couple of key advantages surfaced, mainly:

1. The results are more realistic
2. It allows for unanticipated issues to surface
3. Priceless inputs from the “average Joe”

Below is a copy of the transcript:

Original Article:

Bill Moore of RadioTime shared his experience of gathering feedback from users in a sandwich shop. It’s amazing how easy it is to get user feedback with zero overhead, that’s cheap, and leads to actionable results.

I asked him why didn’t he go to the (in)famous coffee shop since their wi-fi usually works well (wi-fi in the sandwich place was flaky). He said the coffee shop is louder. At the sandwich shop, people stay longer and the tables are bigger.

RadioTime.com offers a free complete guide to radio and sells a product that lets you record radio just like TiVO records TV. When doing on location testing, consider the following:

1. Pay attention to time and setting: People are less likely to be in a hurry at lunchtime than in the morning before work, for example.
2. Make testing clear before starting: Be upfront about time commitment, what you’re doing and why, what the user will do, and any other relevant information. RadioTime lets the user know that there’s nothing to download.
3. Keep notes: Be ready by having a form and a clipboard. Perhaps, consider recording the user’s feedback with a non-intrusive recorder. Use a clipboard in case you can’t access the table when making notes.
4. Talk with the store manager: This assures the store that you’re not doing anything suspicious and plus you’re building a relationship.
5. Wear logo or display company badge: Helps with trust and credibility.

Here’s what Bill said about doing testing in the sandwich shop.

“The information is especially useful to us because users are not in a controlled environment. They have their own PCs with their own media players, shortcuts, browsers, etc. Not surprisingly, this shows a problems that are invisible when testing on our systems.

“I borrowed the idea from a book on Intuit (Inside Intuit) where they had a program ‘follow me home’ watching customers balance their books. Then the famous Krug book Don’t Make Me Think made the case to test early and often on anyone you find.

“I walk up and buy $30-$50 of gift cards and sometimes find the manager to explain. So shop is getting a benefit.

“We can test a half dozen people and as many computer configurations cheaply. Ours is easy because it is a general consumer site and it’s easy to get people interested.

“The setting and time is important. I wear a shirt with our logo and have a clipboard. Mid-morning or afternoon is best when people are less pressed for time and open to do it. In our case saying we are a local company testing a site helps because they have some empathy and won’t need to download any suspicious software.

“Asking for five minutes up front is important, sometimes ten minutes depending on what we are doing. Their patience and the incremental value of their testing fades quickly anyway. Like any testing — up front make it clear we are testing the software, not the user. And invite them to think out loud to better understand what is happening.

“I make a form with the main tasks to make it easy to keep notes. It is impossible to both watch and keep good notes by yourself. On the other hand two people are more intimidating and you can capture the main points later.

“Like any use testing, it can be painful to watch.

“I would like to see more developers go to the shop or watch when we do testing in the office. They are the ones that need it the most to witness first-hand why the details are so critical.”

From a content POV, the feedback is valuable. Users made comments that they didn’t understand what unreliable streaming meant and other similar comments. This helps me identify what needs explaining or rewriting.

My response:

Tell me if I follow correctly: Do you do the test only on users that have their own PC with them? So if you happen to be in a store where nobody has a machine or is willing to take the test, then, tough luck?

Why worry about having the users test in their own environment (PC)? The focus of the test is on usability. Testing to see if the site works properly across different systems should be part of QA work and can be done separately.

Bill Moore’s response:

Depending on the store and the time, you are highly likely to find users working on their notebook. So, no computers has never been a problem. The setting and your approach are critical to get to a sure. Mostly, people are empathetic.

We test with different OS/browsers. But the main advantage of Panera’s (restaurant) testing is getting users on their computer.

This surfaces plenty of bugs that are unlikely to be discovered with “QA testing.” Watching how people react and recover puts an entirely different perspective on which “bugs” are important. We find far more QA problems in an hour compared to days of brain dead “Does every feature work on XP/Firefox? Yes, Vista/IE, yes…”.

Also you find a lot of problems that people won’t report because the situation made them feel stupid or somehow responsible.

Use testing is more realistic than a lab. The user is completely comfortable with their machine. That means testing is on our software in a real-world environment. They don’t stumble, “where is the delete key”, “how do I open the browser again”, “I hate this mouse”, “where did I save that download?”

You can see if they find our shortcut and tray icons amid a cluttered desktop (as compared to a sparkling clean Vista install).

You see problems like the Skype IE “phone number dialer” screws up layout (no one would report that).

All kinds of surprises come up.

Most important for our product, we see how users react to the software that depends on browser and media players installed. A big part of the test “that we consistently flunk” is helping them get the machine ready.

We have tested on our machines. My experience is developers especially don’t like the approach above because it’s not focused and there are too many variables. Precisely. The same was even true with the last design/UX team I had. They wanted to focus on “how does this look in Safari”, and tightly controlling testing of a use case. I agree there is a role for that.

But what I found is our main problems have been outside of an unrealistic starting point “OK from this screen how would you LISTEN to the program.” “they pressed listen, it worked!”

Another use testing trick that worked great for us was recruiting from our building. We setup a table in the lobby with coffee and donuts. Then we had a schedule form and offered $25 for 30 minutes. Everyone was in the building, so it was an elevator ride for them. We had a controlled environment and the developers could watch (but most would find an excuse not to). About 50% are no-show.

We’ve not done any of this in a year and I plan to with the site updates. It takes some time but is priceless.

Meryl’s comments:

We have a few computers in my house and each one works / reacts differently to RadioTime especially where audio players come in. They have variations of browsers and versions installed. One has IE 6 and Firefox 1.x. Another has IE 7 and Firefox 2.0.

What Radiotime has that many typical sites don’t have is Internet audio that relies on the user’s installed media player.

Add browsers to the equation. Windows Media rarely struggles in Internet Explorer, but might make trouble within Firefox. Sometimes Firefox doesn’t handle an already installed plug-in correctly because the user hasn’t installed it from within Firefox. Crazy, but true.

My response:

Bill,

I appreciate how you shared your experience and compared traditional QA with this new and dynamic approach. I now see the value of your sandwich shop method as it contributes a sense of relevance and ownership that is absent in a controlled environment.

Could not help but smile on the Skype phone number dialer. I use Skype and the first thing I do is disable the Firefox and IE plugin. I guess that being in the IT industry, it is inevitable for us to unconsciously “optimize” our systems by removing unnecessary clutter or by not installing them at all — something that everyone else tends to do.

I understand how approach is key to success in getting users to participate. I run a store that sells tea (cold and brewed) and one thing I learned is that atmosphere (ambiance, crew friendliness, store cleanliness) contributes a lot to the customer’s level comfort level, which is relative to their mood levels.

It is interesting that you used the same coffee and donuts setting that I’ve seen being done by television and radio stations to gather viewer response. Based on my experience the gem in this approach to information gathering is not the answers to the survey itself but on what happens in between where the user shares personal views and comments — information that would otherwise not have been anticipated by the in-house team.

Meryl,

You are right on the money on the importance of the installed media player plug-in, and I agree with you. On my work machine I use Firefox 3 Beta 2 as my main browser with Foobar2000 as my default media player.

When a site breaks, most of the time at least I understand why (e.g. Firefox is causing it or I do not have the required plugin) but that is because I am in the IT industry. I have 3 machines at home and one of them is running on Linux. How many others have unorthodox setups? I’m sure there are plenty.

With that said, nothing beats my ultimate personal usability tester — my 60 year-old mom. It takes a lot of patience, but in achieving the usability level of a standard (non-scientific) calculator needs a lot of it. To quote a common pun when comparing operating systems: “it’s so easy even my parents can use it!” Maybe I’ll create a separate post on this some day.

Share This