Conducting Usability Research for Mobile

This article is based on a podcast with Jared Spool and Cyd Harrell. Listen to the podcast or read the full transcript.

Jared: Does mobile usability testing require a lot of hardware?

Cyd: It can. I think there’s a bit of a perception that it requires more than it actually does if we’re going with the traditional lab model. Or we want to take perfect videos and to get every single mobile device that our audience might use all lined up in the lab. Pretty soon, we’re indeed talking about 20 pieces of hardware in addition to what we’d normally have for a test.

But there are ways to minimize it, and there are ways to do it without adding on too much hardware. Think a little bit more about how you might do a guerilla research study or how you might do a study in a home or even how you might do a remote study, which is just starting to become possible with some of the new developments. It’s pretty exciting what’s going on these days.

Jared: Let’s talk about this. If I was doing this in a lab, do I need to retrofit the lab with special cameras to capture things or are there tricks to this?

Cyd: There are some tricks. My favorite way to set up the lab for a mobile study is to have the participant sit in a comfortable armchair and place a little target on the arm of whichever hand they’re most comfortable using. We have an over-the-shoulder camera, which can be something as simple as a flip or one of the small video cameras on a little boom that can focus on the mobile screen.

Then, before you start, do a little bit of tweaking to make sure the device’s brightness is set so it’s easy for the camera to catch it. You can get a really nice video without ten extra pieces of equipment.

Jared: That’s interesting. I’ve heard in some presentations about molding these little plastic sled things that you mount the camera on and you mount the phone on. Or you need to put the phone in a little box on the table that has to stay there so the camera can hit it.

But you’re just doing this in a comfy chair.

Cyd: I think it’s better in a comfy chair, just because the participant is more comfortable. You can still have a target area on the arm of the comfy chair, but rather than necessarily laying the phone down, you can figure out in your pre-flight—which I certainly hope everybody does for a study—what’s the approximate angle that a couple different people hold their phone at. You can set up the boom so that you just need to do a little tweak to the angle as each person comes in and you’re good to go.

I do think sleds can be useful if you want someone to walk around in a lab environment, but they are hard to use out in a real world environment, which is, of course, the other environment where we really want to see people doing mobile testing because of the in-context nature of mobile use.

Jared: Now you’ve done some out in the field studies, right?

Cyd: Oh yes! It’s incredible fun, because you really can get embedded in peoples’ lives. It can be crazy technology-wise. I’ll tell you that the first mobile study I ever did was probably the most technically complex, but in some ways one of the most interesting. A major car manufacturer wanted to create a concept car for the 2009 Geneva auto show that would take account the way drivers use mobile devices in their cars.

They asked us to do some ethnographic research on how people use various devices, including phones, iPods, and anything else that people actually happen to use. We proposed that we would actually accompany participants on trips that they planned to take. Because of the location of the car company that was sponsoring this, we needed to broadcast it to a couple of different countries in real time so that they could send in questions.

This became quite an adventure, but it worked really well. We had a camera person along taking high-res video, and then we had a researcher with a laptop who also rode along and used the webcam on their laptop together with one of those Sprint devices that lets you connect pretty well from anywhere.

These days you’d use WiFi, but that was what was available at the time.

We got some incredibly rich information from seeing the way that they might use devices in a car.

By being there on trips, we could see what their real environment was like. We could see how they manage with the dog in the car? Was it really true that they would never text while the car was moving? (Not true for most people.)

I don’t feel that we have to start with the assumption that in-context mobile research isn’t possible. If we’re choosing to do it in a lab, it should be for good reasons. But it shouldn’t be the choice because we assume that we can’t max out the realism of our research scenarios outside the lab.

Jared: It feels almost like the lab is not necessarily the first choice for doing the research in a lot of these projects. Doing them in context is not that much more difficult. We should be strongly considering that as we’re doing the studies.

Cyd: I think we should. I strongly encourage people to do it. Again, live broadcast is a complexity, but if you’re not doing live broadcast, you have a lot of ways that you can be with someone.

We’re getting better and better options for putting together sets of tools that can let us do things like diary studies or experience sampling studies using people’s mobile phones as essentially the research platform.

Sometimes we’re talking about instances where the mobile phone is also the topic of research. It’s also possible to do it on for different topics where they might be doing an activity episodically that’s not specific to their mobile phone.

Jared: Say a little bit more about that.

Cyd: Well, there’s a famous example from 2008, called the “Happiness Project”. A graduate student at Harvard decided to survey people’s happiness and try to understand what activities actually make people happy and which don’t. He set up a really fun incentive for this. He said, “If you participate in my project for two weeks, I’ll send you a graph of your happiness.”

The study was engineered that you’d receive a text on your iPhone. This, of course, was on second generation iPhones, the 3Gs mainly and some of the first generation ones. When you received the text, you were supposed to respond to a survey in a mobile website that asks, “What are you doing? Who are you with? And how happy are you right at the moment?”

The unfortunate thing was that the iPhones of the this time made filling out a mobile survey enormously painful. It was really hard to get through the survey that was only supposed to take 30 seconds, so a lot of people dropped before two weeks.

They ended up getting a lot of good data, but now, there are some really exciting developments in terms of mobile storytelling and mobile survey and mobile form creation applications, where you can really create a pleasant experience for your participants taking part in your research.

Jared: This gives us an opportunity to use mobile as a research tool to study things that are much broader than the phone itself. Like if I was trying to figure out when people are needing, for instance, my company’s intranet, I could have employees collect that data on the phone throughout the day in a diary-like study.

Cyd: Very much so, yes. We’re at the stage right now where a lot of researchers are duct taping different apps together. There’s one platform that’s really designed for this which is e-Scout. They have a great team over there, and the app is developing along.

But I have heard of people using Survey Gizmo. I have used Blurb Mobile myself. There is a really nice new web and mobile form generation product that’s going into beta called “Typeform” that I haven’t seen yet, but I’m really excited about. Things are really moving fast.

Jared: Well Cyd, thank you so much for taking time to talk to us about this.