Full Transcript

Adam Churchill: Welcome everyone to the SpoolCast. Recently, Cyd Harrell presented a fantastic virtual seminar for the UIE audience. It's called "Mobile Research Techniques Beyond the Basics." The recording of this seminar along with over 180 UX seminars is now part UIE's "All You Can Learn."

Cyd was a highlight for us at each of the past two UX immersion conferences. We had to have her in the virtual seminar program, and she was kind enough to say yes. The reason being she has the inside scoop on how to design and execute mobile research that gets you the most usable data for your money in the lab or out in the field.

A great mobile research gets you more than just your user's answers to your questions. It tells you whether your site, application, or your product, the one that you're building, will actually solve the real-life problems that your users are facing.

In today's podcast, Cyd joins us to talk a bit more about mobile research and getting it right for your users. Cyd, thanks for joining us.

Cyd Harrell: Adam, thanks for having me again.

Adam: For those that weren't with us for your amazing presentation, can you give us a bit of an overview on your virtual seminar?

Cyd: You bet. We are talking today in the context of mobile research having got both more essential and easier in the sense that mobile phone penetration in the United States is now something like 90 percent, and smartphone penetration is something like 58 percent. This means we all have an imperative to look at what our users are doing in the mobile space, and, at the same time, the very basics have, thankfully, gotten easier for us.

Just checking how your site or app looks on a bunch of different phones is no longer the incredible challenge it might have been four or five years ago, but there's more to it. One of the terrifically important things that we think about at Code For America is this notion that mobile is starting to fill the digital divide but do it in a slightly different way than people might have anticipated would happen.

There are something like 75 to 80 percent of Americans, depending on population, who have broadband access at home to the full Web via a computer of some kind. In a lot of places where that's not available to people for one reason or another due to infrastructure, socioeconomic status, affordability, et cetera, mobile phones with smartphone capabilities are filling the gap. That's everything from the latest iPhone to a cheap MetroPCS smartphone with a couple generations back Android operating system and a pay by the month plan.

It's really interesting to note that if we think about making products for everyone we need to address the segment of the population that is coming to the entire Web via that three to five inch screen on a mobile phone.

One of the other really interesting things is it's becoming clearer, as more people look at this, how mobile phones are used somewhat differently from desktops when people have a choice. One of the key things is that people use mobile in a more personal way. These devices are things that we carry in our pockets all the time, and I think most people have a confidence level that this is going to remain private and their own so they will look at things that are more personal that they might not put into a desktop computer.

One of the examples I talked about in my virtual seminar was aids.gov see very different searches coming into them via mobile devices versus the desktop. You're not getting a complete picture of what your users are up to at this point if you're not looking at mobile.

What we talked about a couple of weeks ago when we did this was how do you get past the hard parts, because there are significant challenges with mobile. If you're doing lab studies, you need to figure out how to make participants comfortable so they can act as naturally as possible, and you also need to do all the logistical backup of different backup phones and power cords and how do you record. Talked about my favorite armchair method of seating people so that they're comfortable.

We talked about the challenges of actually getting into the field and working with participants in a native environment like a street or a store, doing intercept recruiting, getting all the commissions necessary to do walk-around research with someone using a mobile device in context.

We also talked about the idea that because of the personal nature, because people carry them everywhere, mobile phones offers an opportunity to do research into deeper questions much more easily than we might have been able to do it before. Doing things like diary studies where we look over a month or two at people's activities around, say, exercise or around eating habits or something that's not even in the health space at all via using their mobile device to repeatedly contact them and have them check-in with us.

Finally, we talked a lot about ways that you can recruit people for mobile research, to recruit people with particular devices but also to reach them in mobile contexts like via Twitter or even via SMS.

I think we covered a lot and had fantastic questions, and I really enjoyed talking to everyone.

Adam: Yeah, it was a really engaged audience, and we did have a ton of great questions in from them. Let's get to some of those. Regarding the longitudinal techniques like, a great example, one you talked about a lot, was a diary study. Some folks wanted to know can you use those same techniques for researching maybe an application that they're designing or a website, something other than a mobile design?

Cyd: Yes, you absolutely can. You can use those techniques to research anything on or off the phone, in a way. You're really taking advantage of the nature of the phone as a connected research platform that's with someone all the time. You could absolutely use it just to research a mobile app that is something that a person might use daily and find out about their usage and have everything photograph that they send you be a screenshot of what they're doing with it that day.

It could be something completely unrelated to phones. The one that I have used a lot is commuting. People who commute in a city via various transit methods, finding out where they're running into barriers where the signage is working for them. Doesn't necessarily have anything to do with the phone itself at all, but the phone is a useful platform.

Those techniques are accessible across all kinds of subject matter areas. The really key point is just to take advantage of the presence of the device and how you can turn it into research platform and how you can make it non-intrusive for someone to send you, over time, various responses related to their experience with a particular thing.

Adam: One of the questions that came in, we talk about the presenters that do things for UIE that speak about mobile design, and they all have pictures of themselves with this array of mobile devices. Thinking about that crazy offering, and one that's changing every day, how many different phone types or mobile devices do you need to have in your sample to know that you're covered for these research studies?

Cyd: This is a great question, and we're going to make a little bit of a distinction between testing and research. We have some great platforms now, things like Ghost Lab where you can look at your app or your site on a whole bunch of devices with a whole bunch of form factors relatively easily. I think of those as more of a QA or testing type of platform.

When I'm doing research and I'm trying to get a little deeper, I have to make some assumptions about where are there actual behavioral differences between people that are using different operating systems or different versions.

The obvious one, of course, we have two really dominant OSs that are used on mobile devices right now. It would be unusual that you wouldn't want to include both iOS and Android in your sample. If you have people who are likely to own tablets then you might, indeed, want to include both tablets and mobile, but you don't have to go too far into micro-segmenting by device. You want to really look for those behavioral differences.

One of the distinctions that I sometimes talk about is also just a crummy phone, having one in your sample. One of the smaller Androids that's not one of Samsung's big, shiny ones. Test with a cracked phone if somebody has a cracked phone. See what that experience is like for somebody that's not able to fix that.

I was asked for a number. Now I'm going to waffle around a little bit and try to come up with one. I would say, most of the time, if you could get three or four types judiciously distributed in your sample -- say, a nice, big iOS device, a nice, big Android device, an older iOS device, and an older, smaller, crummier Android device -- you are probably getting pretty good coverage of the experience, as it stands.

Adam: Very good. Vanessa wanted to know if you had any scrappier techniques that you've tried. She made a comment that some of the techniques seemed a little heavy, especially for teams that are currently doing little to no mobile research at all.

Cyd: Yeah, this is a fantastic question. I'm sorry we didn't get to it as one of my favorite things is designing scrappy research methods. I think she asked this about the part of the presentation where we're talking about techniques for in the lab, and I spent a lot of time talking about how we can record and transmit our sessions as they happen.

My advice, if you're in a truly scrappy setting, is that you can just jettison all of that. Recording and transmitting are really extra, and a tremendous amount of the overhead lab setting is making sure you get those videos right and making sure you have the equipment to position the cameras in different ways and that you have wire connections so that you can send video to observers in real time and all of that stuff.

If you're not doing any of that, if you're just sitting next to somebody in a comfortable chair with a connection that is strong enough for both of you to be online and there are two of you on the team, let's say, and one's sitting right next to the user and one's sitting behind and somebody takes notes, that's truly all you need.

You have to, basically, put the onus on yourself there to take good notes and record what's important about your sessions. There are a few practices you can do where you, in a very disciplined way, write down the three most important things you learned from each session as it concludes, things like that.

You absolutely don't need all the overhead of cameras and microphone booms and a bunch of other things that we talked about during my seminar.

Adam: Sarah wanted to know if you've ever done a study where you're jumping from device to device between, say, a mobile phone and a tablet or an iPad and a desktop?

Cyd: I've done a couple. It's interesting. It's been less X app working on the different platforms and more around a particular task that people do that they might do at various times on different platforms. A lot of the same device would apply.

If you're doing it in a lab setting you want to make sure you have the right backups for all the relevant devices. If you're doing it in a, say, diary study setting where you're asking people to send you photographs you want to figure out if you can one device as a device on which you administer the study and contact the person and remind them I'd like you to send a photo and then use that one to, say, take pictures of the other one.

You can administer a multi-device diary study by having things available by something really common like email where it would be easier for people to send you something from any of the devices.

It is very tough to get cross-device studies done in a lab setting because people's use of different devices is really so tied to the context that they're in, whether it might happen to be their home desk versus their home couch versus out and about.

Adam: A bit more about the diary studies. Let's talk about that for a bit. Jill was looking for some guidelines. How do you decide the right amount of time? Is it one week, two weeks, a month? She was also wondering about the number of participants.

Cyd: That's a great question. The length of time, I think, has to do with the time cycle of the thing that you're observing. When I used the example of something related to commuting and transit, a week might well be enough because then you might talk to someone with two work commutes and five coming home commutes.

My sense is that, generally, three to five cycles of something will be enough, but if it's something like a weekly activity then you'd need to extend that out. If it's something where you're trying to get at something pretty deep or you want to observe, say, the on-boarding period of a particular thing -- I'm thinking of a study that I did a number of years ago which wasn't mobile, but we were looking at people who received a medical diagnosis and who needed to get information about what was going to be happening.

That process was really a couple of months long. If you wanted to observe something specific like that then you'd want to fit your timeline to the timeline in which it actually happens.

In terms of the number of participants, the diary study apps which are available option give you the option of researching a lot more participants and getting a ton of data. It's really limited, mainly, by your capacity to analyze. It's a nice way to get at several different segments. You do end up with something like three or four data points per person per interaction.

You've got to look at what the capacity of your team to take in all of that and do something meaningful with it is.

Much like usability, the minimums are somewhere in that same range of 5 or 6 participants, 10 or 12 being a nice sample, but it does allow you the capacity to look at, say, 40, which is really nice for those of us in the civic space where we're wanting to look at really broad audiences.

Adam: With regard to those diary studies, can we talk about the actual prompts and questions that you're asking those participants? Are they open-ended or can you actually make your questions be a bit more directed?

Cyd: [laughs] They have to fit your research goal. If you're asking about people's feelings it's very likely that open-ended would be good, but you can ask people to rate a sign that they just took a picture of for you, take a picture of the most interesting thing in their space and give you a five word description.

Whatever will actually suit what you're trying to find out is most appropriate there. It offers tremendous scope for creativity, and there isn't an established set of best practices because not that many people are doing it yet. I feel like saying to everyone who's going to listen to this that you will be part of figuring out how best to do this. You should take this as an opportunity to really focus on how this technique can afford you the ability to find out more than you could find out otherwise and share what you learn.

Adam: Cyd, is recruiting for a mobile research study any different than a typical usability study? How is it different? Can you recruit via social media?

Cyd: [laughs] It's slightly different in that I always say the number one thing for recruiting for any usability study is matching the person's task to what you want to study. Someone who fits your demographics but isn't interested in the task that you are setting is never going to be a good match.

In mobile, we have to add another layer where we really are looking at what devices people have or are comfortable with. We may be better, in future years, as we understand more about how particular device choice corresponds to typical behavior and be able to look at that more from a behavioral perspective.

That's the additional layer. One of the things that we think about is how do you then recruit people in a mobile context if you are looking for people who are doing something with their mobile devices? Some of the options absolutely are social media. I've had great success recruiting with Twitter.

It's basically important to cram the same few things on a tweet that you would put on a more lengthy recruiting appeal where you're saying this is what I'm doing, this is how much time it will take, this is why I'm contacting you, and this is how much the incentive will be, if anything. Then you can do things, if you are recruiting via Twitter, like editing your profile that makes it clear your legitimacy to be asking this particular question.

Sometimes I'll edit my profile to make sure UX researcher is the first thing, and then I'll put current study, whatever the study is I'm recruiting for. That tends to really help the response rate.

We actually had a team here at Code For America having some success just recently with an SMS app that they've been working on. This is one of my favorite stories from this year. They had been working on an application for people who receive food stamp benefits to check the balance of their food stamp account via SMS. Previously, in most cases they have to call into a number and wait on hold for a while or they have to visit an office. In some cases there's something on a website, but not very often.

This team created a way to send a text to a particular number and then a couple of minutes later receive back your balance. They wanted to know how this was going. We talked a little bit, and they decided they would take advantage of the two minute gap between when a person submitted the request and when they got the response to ask if people would participate in a short survey via SMS.

They sent them a text that says while you're waiting for your response could we ask you a few questions? If the person agrees then they ask them just a few research questions around before you had this application how did you normally check your balance, do you normally check it at home or at the store, and how is this working for you and would you be willing to participate in future research in person or by phone? That's it.

Adam: This has been great. Thanks for joining us again.

Cyd: It's been my pleasure, Adam.

Adam: To our audience, thanks for listening, and thanks for your support of the UIE virtual seminar program. Goodbye for now.