Episode #10 Facebook Becomes Anti-Social (Part 1)
We discuss dream panels, CUE studies, whether we're an engineering discipline or a craft, the value of heuristic evaluations, and how whether we should learn anything from Facebook's recent loss of face.
[Editor’s Note: This is a transcript of the SpoolCast #2 recording. It has only been “mildly edited” at this point.]
Jared Spool: Welcome, everybody, to our second episode of the SpoolCast. We have today a great crew, a full boat, of the full SpoolCast crew. We have DeWayne Purdy…
DeWayne Purdy: Hey.
Jared: Kyle Pero…
Kyle Pero: Hello.
Jared: Joshua Porter…
Joshua Porter: Hello.
Jared: Lyle Kantrovich…
Lyle Kantrovich: Hello.
Jared: Rashmi Sinha…
Rashmi Sinha: Hi there.
Jared: And Nate Bolt.
Nate Bolt: Hello.
Jared: I think today, I want to start by talking about conferences. During the conference season, which means Fall is conference submissions season, and so we are in the midst of submitting things to conferences. A whole bunch of conferences that I’m involved in, and that many of you are involved in too, have released their requests for submissions, and I was thinking about panels. What makes a good panel, and what would be a panel I’d want to see, and who would I want to see on it. For our audience at home, I last week asked all of our crew members here to talk about what their dream panel would be. Who wants to go first?
Kyle: I’ll go. This is Kyle. What I would want is probably very different, I think, from what most people would want. After going through a bunch of conferences I’m just tired of the typical panels, watching an issue’s gurus sitting around on a stage or a table and debating or discussing the latest topic. I think it’s interesting, I think we learn a lot from it, but there’s something I’m much more interested in hearing, and that’s pretty much what our industry’s clients have to say about us as a profession. I think we need to ask ourselves if we’re growing and improving our service based on what the clients want – or what we think they want – doing a little bit of our own user research I guess. I believe that our clients, and not the industry, should definitely be setting our standards, and I don’t know if we’re doing that. A panel of clients discussing their needs would be quite interesting to me, this is just my opinion. I know every one of us works on different products – some of them offline, some of them online – but basically, the need is the same. Can the products be used easily to accomplish the task? I think if we just take a moment in one of these conferences and just stop listening to each other for a change and start listening to who we are servicing I think it would be pretty interesting and different. That would be my dream panel.
Jared: How would you imagine that working?
Kyle: I know it’s kind of hard logistically, do you get product managers? Do you get people like that, or people who have used the service and what they thought? Would there be a specific service, like user testing? Or would there be something like, “Here’s the evaluations, you can pick a specific service to discuss”? Different people have worked on different products, get them in a room together and see what they liked and disliked. What gaps are we not fulfilling? We go on these assumptions that this is what our clients like; and then even “clients” is debatable. We talked about this on Listserv recently – are we servicing the end user, or are we servicing the people who hire us? Essentially it’s both, so really, we need to be listening to both. I think we do a lot of listening to each other, but I don’t think we do a lot of listening to those two.
Jared: I think that’s very cool. I’ve felt for a long time that we are an echo chamber, and we just talk to ourselves a lot, and I think that gets reflected in some of the things that I see, of folks not being very connected to the people they’re serving. I’ve often thought that we don’t really focus, for instance, on the usability of usability services very often.
Kyle: It’s ironic, isn’t it, that we have these conferences that are all about the user and how we service them, and basically how we built some things that are more usable, but they don’t include users. We don’t include users, or people who hire us to make these products usable.
Jared: It’s hard to sometimes get those folks to speak out, I’ve noticed. It’d be nice to see them on a panel.
Kyle: Yeah, it might be something that could never happen, it might be far-fetched, but in a perfect world, that’s probably what I would want to be seeing.
Nate: At the Web 2.0 conference last year there was an interesting panel that your comment reminded me of, Kyle. They had a panel with four or five teenagers on it, and it was a discussion about how the teenagers live their lives with technology.
Nate: That was all anyone could talk about after the panel, it was “Did you hear that? Did you hear the way that they’re living their lives?” It was completely interesting. They would say things like “I spend my lunch money on ring tones, ” and things like that, and it was great.
Lyle Kantrovich: Yeah, I heard that podcast was great.
Kyle: I have to check that out. Even just sharing clips that we get from user research that we do – I have some really great sound bites that I really can’t share because of confidentiality and things like that. But if we could, it’d be pretty cool, even if we couldn’t talk to the actual end user and get them up there, but just to share the research that we’ve done. I like case studies, that’s just me, I really like to see the birth and death of a project, and see what they did right and wrong, and what they learned from it. I think they’re really interesting.
Jared: Cool, who else has a panel?
Lyle: I have a couple of them actually. This is Lyle. I’d like to hear Jesus, Mohammed, and Eric Clapton. All discussing which came first, the chicken or the egg.
Nate: I think they had that on the Daily Show the other day.
Lyle: They already did that one, yeah. On a more serious note, I’d like to hear Jared, Rolf Molich, Jakob Nielsen, Steve Krug, and Jeff Farrell all discuss how many users do you really need to test with.
Jared: No, you really don’t want to hear that.
Lyle: Not because I need enlightenment, it’d just be a great throw-down, you know what I mean?
Kyle: Everybody loves a debate.
Jared: Oh God, I hate the “how many users” question.
Lyle: No, it’s so tired, but it’s great.
Jared: That is so 1998.
DeWayne: Well I think you have Jakob could also have a great discussion about the importance of the home page, too, Jared. That could be a fun aspect of that one too.
Lyle: Jakob’s got a whole book on it.
Jared: Yes, yes he does. I’ve actually invited Jakob, and I think other people have invited Jakob, to be on panels, too, and he has always refused.
Lyle: Oh, really?
Jared: Mm-hmm. I don’t know why. I’m not even going to venture to hazard a guess.
Lyle: I’d also like to see a panel with Rolf Molich and then Jakob discussing the merits of heuristic evaluation.
Jared: Right, because Rolf has really changed his position.
Lyle: Yeah, they co-developed it, and I took a tutorial at CHI a number of years back from Rolf, on usability testing, and he was pretty cautionary about using heuristic evaluations at all. While I still do expert reviews and think it a valid method, he gave some really good points that people should think about when considering to use that method of testing.
Rashmi Sinha: What were his points? I’m not really familiar with what his position on heuristic evaluations is.
Lyle: This is a number of years ago, so I’ll probably short-sell it, but one of the key points was that we have “experts” who give their opinions, and present them as facts. The key thing I got from him was that we need to be really clear about when we’re stating fact, and when we’re stating opinion, and we tend to get real muddy about that.
Jared: I think that’s a piece of it, I think that the other thing that he’s come to the conclusion of – at least what I’ve heard him say – is that the heuristic evaluation doesn’t take into account the domain expertise of the person doing the evaluation.
Lyle: Yep, that was another one.
Jared: And, if I were, for instance, to evaluate a system for surgical neurology, chances are none of the recommendations that I could potentially come up with are going to improve the system, because I don’t have the expertise. I could say there are general principals that I could adhere to, but I’m not in an operating room, using this equipment under operating room conditions, nor have I ever been in one. So many the things the system does, they do for a reason, that you wouldn’t do in an ATM kiosk. So if you were to follow the recommendations I do make, there’s a good chance that you’re not going to improve the system because you’re going to be changing things you don’t need to change. You in fact may break things. There are going to be all kinds of things that I’m going to miss because I don’t have the domain expertise to know that something is wrong, or in the wrong position, or stated poorly, or confusing, because it’s all Greek to me.
Lyle: Yeah, and are all studies Greek? CUE studies, whose method people aren’t familiar with, those compare usability evaluation studies. I’ve done at least four of them now.
DeWayne: Aren’t we on six or seven?
Jared: We are doing number six this year, at the UI conference. Kyle was a participant in number five, if I remember correctly.
Kyle: Yep, we tested Ikea, it was amazing.
Lyle: So these are great studies, and one of the main findings is we don’t have consistent agreement between evaluator to evaluator.
Rashmi: And what is the problem with that? I’ve never really seen the point of those studies. I’ve been a big skeptic. I haven’t really ever spoken up about this issue ever, but I don’t get those studies, I don’t really get what is the big problem. It’s a way of solving a problem, and professional people are going to see different things. So people who buy into those, maybe they can explain that to me.
DeWayne: Well, what they’re trying to do with them is show that, if there’s a field that’s ostensibly increasing the usability for a large population then theoretically it should be some of the same recommendations to make the usability easier for a large population. So, if different people are suggesting different things then maybe it’s not as scientific as it could be.
Lyle: It’s basically just finding the problems, right? So if five teams can evaluate the same website, you may get a very small amount of problems that are found by every team.
Rashmi: But anyone who thinks that this is science, hasn’t done science! I have done science, and this ain’t science, by any stretch of the imagination. I disagree with the whole notion of trying to make it scientific because it isn’t scientific. Second, you have a big problem…
Jared: Making what scientific? The study itself, or the field?
Rashmi: The field is not scientific, and now people have this big notion from experimental psychology, but really it’s not. Experiments have – there’s a way of doing them, and it’s very hard to do that in the field.
Jared: So, you have MindCanvas, right?
Jared: And MindCanvas takes results and presents them to the design team, with the intention that the design team’s going to act on those findings.
Jared: OK. If MindCanvas grows, and you have – because MindCanvas as I understand it today is not an automated analysis, right? You collect data, but the actual analysis is still done by your team of professionals, right?
Rashmi: It’s kind of mostly automated. There are a few labels we decide which ones to push. It’s heading in the direction of automation. Of at least for the self-service being fully automated and if you want to do analysis right then.
Nate: It’s pretty automated though. Isn’t the stuff that you guys do actually more just like getting the data ready?
Rashmi: So, if for example, you’re doing the analysis for open sort, we look at three different algorithms for doing the clustering and maybe make a call as to which of the clustering algorithms is appropriate in that situation. So, you know, it’s choosing from maybe a package of pre-chosen solutions that we typically use. So, yeah, it’s mostly automated and what we’re doing is for the self-service part we are taking the best solution and just saying this will always be done.
Jared: Ok. But you have people interpreting this data?
Jared: Ok. So if you had four different teams, that were out because your volume required that you have four different groups interpreting data, and they’re each giving whatever clients they particularly serve, each team is giving their analysis of the data to them. Now, if you were to find out, if you had all four teams analyze the same data and they come up with almost no overlap in their results, would you be happy with that?
Rashmi: So first of all, one of my basic problems, yes that would be a problem. To some degree, I mean, the solutions they found they would be different I would expect. So going back to the Rolf Molich study, one of the problems that I had with that is that he buckets the usability problems and the bucketing is not consistent. So I remember actually playing around with the data when I was teaching at UC Berkeley and I was like, “Well, the number of problems and the overlap in problems that you find completely depends on how you bucket these problems.”
Jared: Yeah, but we’re not talking about that, we’re talking about how you don’t think that the profession, the act of conducting these studies, is scientific, right? Let’s acknowledge that there…
Rashmi: Yes, I have never said MindCanvas does scientific research, you could use it for science.
Jared: I understand, but if it matters which team I hire, of your four teams, in order to get the results that are going to be best for me, I mean, you’re not pitching it that way, right? You’re going to pitch it that it doesn’t matter which team I hire but each team produces different results.
Rashmi: Ok, so it’s a good question. I would say that we have come up with a particular way, I mean Mind Canvas the way it gathers data probably reflects a lot of the ways I think about data gathering. And if, say, Nate had designed a way of gathering data in that manner maybe it would have taken a different way of gathering that data. So, at a certain level, if you use MindCanvas, you’re probably saying, “Well, this is kind of a rushed perspective on gathering data. And if you had five different teams design five different ways of gathering data, they might come up with different data sets.
Jared: So I’m saying, even if you had one team who took. If you had all four teams look at the same data collection, the same raw data, and let’s say, for whatever reason, they produced different end results for the clients, they had produced different interpretations, different reports, different whatever that would not be an acceptable result in your mind.
Rashmi: I think it is. I’ve seen different teams, the way they react to, you know, I look at every analysis that goes through. And you know, though sometimes, at least technically speaking you’re getting similar levels of accuracy, different teams seem to react in completely different ways and I expect that the different design solutions that emerge out of the same data set. You know, because there is the designer who is applying their creativity at that point to find the best solution. Which is why we made these visualizations so that you let the designer say, “This is the data,” and then they run with it and take it the best way forward.
Jared: But this brings us back to what Kyle Pero were saying. Because, I think, what is a key element of this is that the client doesn’t realize that they have different results depending on which team comes through. We’re not pitching it that way. Let me put it another way. When Ralph ran CUE-4, he had 17 teams look at, I’m trying to remember, oh it was…
Kyle: Hotel reservations.
Jared: Hotel reservations system, the U Penn system. Not the Penn, but the Hotel Pennsylvania or whatever it was, I don’t remember what the hotel was. But he had people look at this hotel thing that was done by the iHotelier folks, and the 17 teams, approximately half of them, did heuristic evaluations, half of them just happened to work out to do usability tests. They found, across all the teams, they found 61 errors, 61 problems with the design, that the client, the iHotelier people, thought were critical designs, critical problems. Things that they said, these are things we definitely have to fix. But each one of those 61 problems was only reported by one team. So in order to have collected all 61 problems, if the iHotelier people, Jim Whitney and the folks at Webvertising, were to hire a group to do it, they’d have to hire all seventeen teams to get those results.
Lyle: Well, this is why I think those studies are pretty interesting: I think that, by doing these kinds of studies, we will learn what helps us find more problems, what helps us find more critical problems, what we mean by those things. But also, frankly, if you’re iHotelier and you hire only one team, and you find, say, ten critical problems, aren’t you still in a better place? So I don’t think it brings in a question of value or the validity of these studies; but these will help us refine our art, our craft. I don’t think it’s a science: I think what we do is craft; it’s part science and part art. I think it gets better.
Kyle: What the studies really show is that it really does matter whom you hire.
Kyle: This isn’t an exact science. It doesn’t matter whom you hire: you’re going to get the same results.
Lyle: And so should you do an extra review of a neurological-surgery system? If you have no domain knowledge, heck no. I think that’s pretty clear. So expert review should truly involve an expert, who presumably understands a bit about the tasks being conducted and a little bit about the domain. And I can probably find problems with your radio buttons pretty easily. If you just have low-level issues. But they’re [not?] usually the ones that are going to change your business.
Jared: Right. I’m not suggesting that the field is scientific. However, I do believe that the field promotes itself as being scientific, and that we say that it doesn’t matter who actually does the evaluations.
Lyle: I would say that we promote the field as being based on science.
Jared: No, no because nowhere in the field do we say that it matters which one of us you hire. Nowhere in the field do we give people any help or techniques on how you because, if it’s craft, then the way you sell yourself is on your portfolio.
Lyle: Well, I don’t know. My company, when we put a proposal together, we talk about what differentiates us from other people in the field. And I would think our competitors do the same.
Jared: But do you, for instance, say which of your actual consultants this is an HFI you’re talking about, right?
Lyle: Well, yeah. Actually, our clients often do care about our consultants and what domain knowledge they bring to the table.
Jared: But do you also talk about which expertise they have and which one happens to get the best results? I mean, do you actually do have you done anything like a CUE study internally, where you actually can tell which ones do better work? I mean, if it’s, in fact, a craft, then some people are better at this than others.
Lyle: Well, O.K. Let’s go to something that’s really thought of as science. Do you go to your doctor and say, “Show me how you rank against your colleagues”?
Jared: Well, in fact, Medicare is doing that now. Medicare is actually rating doctors against each other, so Medicare patients it’s part of a legislative act that came through a couple years ago.
Lyle: God help the doctors who are rated at the top.
Jared: Right? Or the ones that are rated at the bottom. But they actually are rating doctors on a variety of scales, and they’re rating the institutions they work for on a variety of scales. So now there’s public information out there that says whether a hospital happens to have doctors that are good on these scales or not good on these scales. And whether you agree with the scales or not is up to the individual to decide; but it, all of a sudden, gives new information into it. And where, as a field, are we talking about that sort of thing?
DeWayne: That kind of thing is nice; but it always makes me a little bit nervous. We see that in the higher-ed field as well, where you go online and rate your professor. Well, that may be a realistic rating, or it may just be a reflection of what grade you got and whether you liked the class or not, without any real tangible thought into whether or not the faculty member is a good instructor or the doctor is good. He may be like Dr. House on TV, who’s a real jerk but really knows his stuff. So how do you rate him? Do you rate him good because he helped you or rate him bad because he was rude? So there’s not much science there either.
Jared: No. I’m saying this has to do with craft, right? And so all I’m saying is that, in this field, nowhere will you find it written that it matters which usability person or which interaction designer or which information architect you hire. So we talk about it as if they’re all equal and you just need to get one.
Lyle: No, I don’t agree with that, Jared. How often or where it’s written, I don’t think matters. I think there are books about putting together teams, about getting the right people on a team. And I don’t think it’s any different from having the right people in your marketing department. It’s all about talent and expertise and skill.
DeWayne: And some of the writing out there would lead you to believe that it’s not just about writing your talent or skills, but that you do it this way and it is scientific. Going back to your favorite, Jared, “How many users does it take to conduct a valid test?” You see writings that say that eight users will give you eighty percent of your problems, which tends to lead you to believe that that’s a scientific fact.
Jared: Yeah. There’s a lot of science to say that that’s not a fact.
DeWayne: Well, exactly. But, especially for someone in my position, where I do a lot of different things, my usability is only one of the hats that I wear for people trying to get into it, who are out reading things like that they’re led to believe that there is more science than craft to the practice.
Joshua: I wonder if we got Kyle Pero’s panel together and we actually asked people how they think they hire these folks…
Rashmi: Yeah, that would be…
Nate: Yeah, that’s true. It’s interesting talking to different companies about their evaluation methods, that’s for sure.
Jared: Yeah, I mean, Nate, what do you find when you’re off pitching Ethnio?
Nate: Well, Ethnio’s a little different from our service side, cause it’s somewhat of a product, so I think the sales process or the evaluation process for our clients is a little different. But on the service side, that’s been the most interesting especially seeing a lot of it like any kind of consulting things is going to be personal relationships. But outside of that, I think, some companies have a laundry list of like, “You must be able to demonstrate expertise in this area, and you must have knowledge of this, and you must have knowledge of that.” And it’s not necessarily, I don’t think it’s correlated to size of organization on how diligent people are about establishing their criteria for hiring usability people. Sometimes it seems like people just take the first people they call, other people you know definitely interview like five usability firms. In my experience, nonprofits seems to be the most, foundations and nonprofits seem to be the most diligent for whatever reason, I have no idea why. And that could just be my Bay-area biased experience. But then the product side has been totally different. The criteria for that seems more like just buying software.
Kyle: You know, Nate, this is Kyle, I’m reading a great book now called “Selling the Invisible” by Harry Beckwith. It’s been out forever, since ’97, and I’m finally reading it after about five different people recommended it to me, and he really covers this well. I mean, as a service, they’re hiring you. You’re selling yourself. And I find that I have about 20 minutes, if that, on a phone, and someone calling me out of the blue or emailing me to set up a call and I have 20 minutes to impress them and get them to feel comfortable with me. It’s not about how much I know I find, or anything like that, it’s about the relationship that I can build in that 20 minutes that makes them feel comfortable with me. I think they want to hear a little bit about the service I offer, but more or less I think they’re just feeling you out as a company and if you’re going to be someone who just is high on yourself, and you’re going to sit in their office and you’re going to be on the phone the entire time, or if you’re going to be taking their needs into consideration and actually giving them a lot of personal attention.
Nate: Totally. The personal relationship side.
Lyle: I think that’s totally true. That would probably span a lot of different kinds of consulting.
Lyle: But I wonder if there’s actually, I was just thinking, I wonder if there’s things that are different. That’s definitely going to be the most number one thing. I don’t know if you guys all agree, but personal relationships probably or working relationships are going to be the highest criteria for anybody in choosing of usability consultant or firm. But after that I wonder if it’s different. I have no idea.
Rashmi: I would agree that the personal relationship and the working relationship and how you anticipate, you know, whether you’ll be able to work together, that is, I find that is the most important thing in consulting.
Jared: OK, so going back to the original topic here, which was to put together a dream panel. Lyle Kantrovich, is this a panel that’s interesting to you? And how would you…
Lyle: I think we’re having a dream panel right now, so. [laughter] Debating whether or not we’re scientific or not.
Jared: Are we the right people to have on this panel?
Lyle: What’s that?
Jared: Are we the right people to have on this panel?
Lyle: Oh, I am, for sure. [laughs] I’m joking..
Nate: Nice. My dream panel is.. [laughing]
Lyle: Is me, yeah. Me and all you guys. And a cake, actually. I don’t know, I mean, who would we want to talk to about what is scientific? I mean, to your point, Jared, if we’re just an echo chamber as a field, who would help us from outside of our field judge whether we’re truly scientific or not and what that means?
Kyle: The people who buy our services. The people who invest in us.
Lyle: It’s sort of like going to your car guy and asking him if your doctor’s good.
Kyle: No it isn’t. Not at all.
Jared: Is that the only place it matters? Does it matter? Who else cares other than our clients?
Lyle: Well I mean our clients may help us determine whether or not we promote ourselves in a particular way, but I’m asking the question of who would help us determine if we’re truly scientific in our approach. Or do we just concede that we’re not?
Kyle: Honestly, I would concede that we’re not. Just like a plastic surgeon is not totally scientific. It’s not a science. It’s not, “Do x, y, and z.” There is some judgment that has to go into each thing. Each thing is customized.
Lyle: It’s possible that we all agree on this panel. Maybe there are other people in the industry that would disagree.
Jared: I’ve gone around the country and talked about this topic. I talked about it at CHI. I’ve talked about it in San Francisco at the Bay CHI. I’ve talked about it in Seattle at the Seattle CHI group, and every time I bring it up, every time I start talking about this, everybody steps up and says, “but we’re not scientific, it’s an art, it’s a craft. It’s…”. There’s really a personal part of this. And then I bring up the whole, “OK, I’m good with that.” And usually my next line is, “OK, so, it’s a craft.” And I start talking about, well, if it’s a craft then we need to talk about it from the perspective of a craft, and craft development, and professional development in a craft service which is a very different way that you develop professionals than you develop professionals in an engineering service because they’re very different things. And people get really sort of frustrated by this conversation.
Rashmi: In what way?
Jared: They really think that… I think because it doesn’t have an easy solution, and they’re sort of stuck in the middle of it.
Lyle: Well to me, it’s analogous to modern woodworking. Right? You can call that a craft, and there’s a lot of artisanship that goes into that, but there’s a heck of a lot of engineering that goes into modern woodworking as well. I think we’re similar in that way, we have engineer driven and parts that are artistically driven.
Jared: But modern woodworking, right, if I go to Home Depot and I buy wooden cabinets…
Lyle: That’s not woodworking. [laughing] That’s consumerism.
Jared: But that’s what I think a lot of the consumers are buying. I think we mislead them in many ways, because they think they’re going to Home Depot to buy cabinets. Now, we actually did a study years ago where we actually followed craftsmen around. So I got to watch professional woodworkers actually do woodwork. And I got to watch them use their tools, and all this stuff. So I’ve had some experience watching these folks do this sort of thing. And what’s really stunning about it is that they will tell you that that stuff isn’t woodwork, that that stuff is just made in a factory, it’s not at all the same thing. But the customers who buy it, only a small percentage understand that the stuff they’re buying is different. Only a small percentage can appreciate the difference and it’s the big struggle of those craftspeople to promote the difference. But I don’t see us promoting the difference, I don’t see us showing what is different.
Nate: I think the process is part of it, Jared. That’s what’s interesting about what we do, at least just on the testing side. I mean, outside the expert review side of things. But, watching a few people use a computer and then reporting on where they ran into problems. And maybe suggesting what you can do to fix it, there’s a bajillion different ways to slice it up and do it but that part of it is pretty rote. I’m not sure but I wonder how much that’s really changed. That seems pretty much to be the core.
Lyle: I guess it depends on what you’re talking about as far as scope. Are you talking about evaluation like usability testing and expert reviews, those kinds of things, are you talking about UI design?
Nate: Just on the usability side.
Lyle: Yeah, because I mean, to me, on the UI design front, if you buy, pick your favorite or least favorite ERP system and install it out of the box, that’s like going out and buying cabinets off the shelf. You know, that’s your UI design out of the box. But I rarely have a client that is asking for an out of the box kind of UI design, they all want something custom.
Jared: Yeah but, do they know what about it is custom?
Lyle: Yeah, yeah. I mean, they articulate things like, you know, they want the essence of their brand. They see their customers and their market as being unique. They see a difference in the way they conduct business, in the way their users work.
Jared: So you’re never in a situation where you have to tell them, for instance, that they are not paying attention to the right people or they’re not looking for the right stuff, because they know all that.
Lyle: No I guess I’m saying that they recognize that they need a custom solution. So no one comes to me and says, “Where’s your menu, cause I want to figure out whether I need the number one or number two package.” From a UI design perspective. They’re not looking for the out of the box shopping cart or the out of the box intranet. They all want it custom designed. Because they see the uniqueness of the environment. I mean, am I the only one that sees that?
Rashmi: Well let me tell you, as a consultant I would agree with you that people who come to a consulting company are generally looking for something more custom, you know, tailored to their needs. But there’s a chance that there’s a whole lot of business we’re all missing, because there are people out there who are just looking for things out of the box, and you know, just go with templates et cetera, I guess.
Nate: Well, how many of you guys do end up doing UI design stuff outside of any kind of UX research? Out of curiosity, I mean like, on your projects? Do you guys do any design or UI stuff without being part of some usability testing or something?
Nate: Oh, OK. So, I guess, when I say we do a lot, we do a lot of user centered design that involves usability work. So we’re always doing up-front user research. We’re doing prototyping and testing along the way. That’s a typical design project for us, we do a lot of those.
Kyle: I get those.
Nate: And I know a lot of people in the field who do that.
Kyle: I get people who will get me too late in the game, and they need something, and they need that path through someone’s hands to look at the front-end design of it. But, yet, they’ve put no money or budget or anything like that into or schedule into doing any kind of user-centered design. And of course when I’m brought on board I try to upsell them, but again, you’re brought on too late and all you can do is do the best you can on the design based on their requirements. And it’s the same old story all over again. Brought in too late.
Nate: Cause that side of the fence is way more art and craft, you know, when you’re on the design side. But just gathering the research, I don’t know for sure, but I wonder if that’s slightly more process driven.
Kyle: I don’t know. I think, nothing is ordered like, “I need some form design, how much does that cost?” It’s all custom. I don’t see any of this being commoditized in that way.
Nate: Well, here’s another way I sometimes put this question out is, most people in the field would say that anybody can’t do design, or well. Anybody could do design for sure but for some complex interface or Enterprise software or something like ERP stuff, not just anybody can pick up a mouse and design something that will work. But, I wonder if you guys, or we, think that anybody can do usability testing, or if there’s that perception.
Jared: That come back to this craft vs. engineering thing. Because if it’s craft then you say “anybody could do it but some people do it much better than others”. And if it’s engineering then you say “anybody who has got the basic training will product the same results”, which is why the CUE studies are important.
Kyle: Well, think about is. Our prospects are our biggest competitors. We have to always fight against people not hiring usability professionals saying, “hey, I’ll do it myself”. That’s fine, that’s up to them.
Nate: I think it would be fun to get a second grade class in CUE to do the usability tests, see what they find, or something.
Lyle: I think it’s analogous to “can anybody give a presentation?” Sure, but there is a difference between an amateur and a professional in a speech class, a professional speaker.
Jared: Then why is it that conferences don’t look at presentation skills when they…
Lyle: You host a conference, what do you do?
Jared: We do look at presentation skills, but I’m talking about the ones that I submit to or the ones that I used to submit to and don’t submit to any more.
Rashmi: So I would say that’s because they do the process of the blind review, which is what you are objecting to, right Jared?
Jared: I’m not a big fan of blind review.
Rashmi: I really believe in it. Because I look out on the web, and I look at the kind of general peer review and people commenting on each other and things rising to the top, and I think, “That’s great, that’s wonderful”. But I really miss blind review. Because blind review is a process when you put on that hat, saying “I’m going to review for CHI or for any other conference where they do blind review,” you say, “Well, as human beings we are biased, we tend to pay attention to people we like, we respect, or people who are coming from good institutions. But for this little time I’m going to just judge this piece of work on its merit.” And you definitely feel the pressure of judging something on its merit, and you also feel this intensive curiosity to know who it is and you do Google searches to find out. But still, there is that moment of saying, “yes, we are biased, and let’s try to give new work, new people, who don’t have any history, a chance to rise up.” I really like that, it’s a wonderful example of the way science operates, of giving new, unknown people a chance. And that’s why I really like the process of blind review.
Nate: Maybe we can do blind reviews of people’s presentation skills, too.
Jared: Well, that’s the question. There are two underlying assumptions to what you were just saying. The first one is that blind reviews can’t take into account presentation skills, and I’m not sure that’s true. The second is that blind review is always applicable in all situations.
Lyle: No, I don’t think that’s true. If you are going to see a variety show it’s all about entertaining you, it’s mostly about presentation skills. That’s not what I would expect from a conference.
Jared: I’m a big fan of blind review. Malcolm Gladwell tells a story about a musician, a violinist, I think, a female violinist who happened to have a name that was sort of gender-neutral and got invited to perform at – I’m not remembering this very well – I think it was the Berlin Orchestra. She got invited to audition for the Berlin Orchestra and they had decided for the first time to do a blind performance situation where they set up a screen, and she performed behind the screen, and the director of the orchestra and everybody else listened to her from behind the screen. She was hands-down the winner, and after her performance she went back behind the screen and downstairs to the waiting room. She packed up her stuff, thinking that she didn’t do very well, and as she was walking out the director came running down the stairs and called out her name. She walked up, and he was just about to tell her that she had the job when he noticed she was a woman. His jaw just dropped and said, “Oh my God, it was the first time that I had a woman perform”, because there was this general perception that women just didn’t perform in symphonies as well as men. It turned out to be completely false, and now everybody does blind review.
Jared: I probably just killed that story, he can tell it much better than I can recite it. I’m all for blind review for certain types of content, absolutely for making sure that something is scientifically relevant. But if part of the goal of the conference is to educate – and not all conferences are for advancing the science, and not all portions of all conferences are for advancing the science – conferences like CHI or UPA or ASYS or any of these other conferences, part of them is for advancing the science, but part of them is to educate the attendees who are coming. And I don’t understand why education skills and training skills and those things are not taken into account for those portions of the program. There is nothing that says you can’t designate if the goal is to make sure that you get new voices and it’s not the old mafia crowd coming back and saying the same things, which I’m all for. We had that issue; we have a quota. Every year we have to have three speaker that we never had at the conference before. And I have to tell you, it takes us 18 months to find those people. We are already preparing for the 2007 conference, we are already working on our presentation for 2007, because it takes us 18 months to find the people we want to have. It’s because we have a quota. We don’t deviate from that quota at all. There are ways to get around this that don’t require blind review.
Rashmi: I’d be fine with some kind of approach where you had a few slots for people who had done extremely well in the past and maybe in the tutorials or something of the sort it does make sense. But I think explicitly adding in at least a phase of acknowledgement, that we are ourselves biased, it helps the process and I don’t think it makes us defiant or anything.
Jared: Well, that’s fine. There’s conferences I’ve been rejected from that claim they have blind review. There was one conference that I was rejected from that claimed they had a blind review process and I found it fascinating that the five submissions that I made all were rejected, yet that same year every member of the program committee had something on the program. They swore, “Well, it was blind review, nobody knew.” It’s fine; the conference doesn’t want to here from us. We don’t have to submit anything to them. I’m good with that.
Jared: But it was just interesting to me that that’s the way they went about it. That’s just my take. I think blind review is used in a lot of places as a hammer that is just way inappropriate for what they’re trying to do. Ok, I killed that topic.
Rashmi: Let’s talk about the Facebook.
Nate: I think killed is strong.
Jared: You want to talk about Facebook?
Lyle: Who all from Dream Panel haven’t we heard about yet?
Jared: Is there another Dream Panel we haven’t heard about?
Lyle: Dream Panels. Dreaming Panels.
Nate: Darwin and God.
Jared: I assume Darwin thinks they’re both on the same side.
Nate: Well I was just thinking about the discussion we were just having and it almost sounded like there was an implicit assumption that if you were a scientist that you follow certain rules and that you should come up with the same results as everyone else. But that would also lead to the idea that all scientists are created equal and we know that that’s not true because scientists like Darwin and Kepler, Galileo, they all had something that no one else had. They had this sort of insight to translate what they were seeing and the math behind it into something new. I would just add to that, before we ultimately kill it, I would just add that all scientists aren’t created equal either and we really can’t get away from the person that’s doing the process in the discussion that we’re having.
Rashmi: Correct. When you teach courses and you do even basic scientific experiments and you have every student in the class replicate that, you do find deviations. They might be deviations around some kind of thing that might form a normal distribution that there’s this one most likely result and then there’s deviations from that. But the method and the way that people carry out the method is different every where. And also in science there’s an explicit concept of the meta kind of research where you look a bunch of the different studies on a topic and you make some logical conclusions. So that once again acknowledges that not every result done addressing the same exact question is going to come up with exactly the same answers.
Jared: Ok, so now I’m going to go back to the hiring manager’s perspective, which says, “Where do we tell them, where do we tell the hiring manager that they can expect variations?” And that depending on who they hire they’re going to see some variations and that if they really want to do the job right they have to hire 17 folks?
Nate: But that’s also assuming that there’s a perfect design and that you need 17 teams and that if you hire those 17 teams you’ll come up with some sort of perfect end result.
Jared: No, all I’m saying is that Jim Whitney at Webvertising said that those 61 problems were all very important problems. So if he wanted to find, now it’s an assumption that he wanted to find all the really important problems to fix, right? So, given that assumption, which I think that’s an important assumption, given that assumption, to get that, and the thing about the CUE-4 stuff study was that the people who participated, not to diminish any of the other CUE studies, but the people who participated were like this who’s who of usability people. They were people, many of whom I’ve personally worked with, or known for years and love their work, sitting in on the review session from the meeting, there was a workshop that they all attended. With just this incredible who’s who and these were top-notch people whose work I respect tremendously and yet there was this huge diversity.
Lyle: Jared, I guess I don’t agree that hiring managers are that naive and wet behind the ears. I think that people who are going to shell out money for a consultant or for an employee know that there is variability across humans. They need to figure out, if they don’t know already, what makes them different from one another.
Jared: I understand that.
Lyle: Not just like there’s a skill scale that you’re a seven and I’m an eight.
Jared: But we barely are good enough at communicating why they should hire us in the first place. Let alone, how they should decide which one.
Kyle: Well Jared, have my panel, at 2007. Please.
Jared: I think it’s a great idea for a panel. I think it’s an excellent idea.
Kyle: I think it’s the only way we’ll find an answer.
Rashmi: You should submit it. Is there time still or is…
Jared: Yeah for CHI 2007, October 31st is the deadline. October 20th.
Lyle: For what now?
Jared: For panels. Actually, they’re not panels anymore.
Lyle: Oh right.
Jared: They are now, what? Interactive sessions? Not to be confused with interactivity which is a completely different submission.
Nate: What about “unpanels,” is there such a thing as that?
Jared: Unpanels? Well they do have a format that they recommend which is that you have some sort of bizarre tag pin thing where every few minutes you replace a panel member with a person from the audience.
DeWayne: That could be fun.
Kyle: Could be crazy too.
Jared: It would be a good performance. I don’t know if it would actually produce something more than you’d get on “Whose Line is it Anyway”?
Kyle: How come as a profession, we all seem to have shoemaker’s kid syndrome? When it comes to designing our conferences, our websites, it’s all unusable. Any one notice that?
Lyle: I think right now people who are putting together the conferences are trying to figure out what’s next. We’ve done conferences kind of the same way for 20 years. What’s the new format? So I think people are experimenting a little bit. So some ideas will get tossed out, old ideas over kept and some new things will creep in.
Kyle: We’re just really poor at organizing around things, I think. Just look at some of our names, our acronyms that are ridiculous.
Nate: Isn’t that the hallmark of a mature field though? Ridiculous acronyms?
Jared: I always say that human factors and usability people just produce the worst names for things. It’s been my experience. They also produce the worst charts.
Nate: The worst charts!
Jared: Oh yeah! Some of them are just absolutely outrageous. But this might take on things. Hey I want to change the subject a little because I do want to talk about what’s been happening with Facebook this week, because I think that’s a really big important story that’s coming down. So for those of you who somehow were under a rock and didn’t hear about what’s going on with Facebook. Facebook is a system like MySpace where people, primarily college students I believe though, I believe it’s now open to others but for most part, most of the subscribers are college students. They can basically connect to each other, designate people as friends and communicate amongst each other, and it’s fairly popular. So one report I read said, I think in the New York Times, they have nine million subscribers, so that I would consider as a popular service. And this particular week, they announced a new feature, and the new feature is a feed that lets you basically connect up and see what’s going on with all your friends. So at first it sounds like a good idea, so you have all these friends and get this feed that says that this person added this other person as a friend. But apparently, people are private information that they weren’t expecting other people to see into this system and all of a sudden now the feed propagates this stuff to all their friends.
Jared: There actually is a really good video that’s done by an organization called MoBuzz. I don’t know anything about them. They put together a little four and half minute video that talks about the Facebook issue and it really sort of sums it up, which is that basically people are finding this creepy, that other people can see the personal information that they’re entering. But the MoBuzz people say, “Well, we think it’s creepy that you’re entering all this personal information to begin with.” So there’s been this backlash, and the part that interests me is not so much the privacy part, the privacy part is interesting, but a lot of people are talking about the privacy part. The part that hardly anybody is talking about is the design process and what happens here in terms of Facebook added this new feature and the users didn’t like it and they revolted, I mean people asked for boycott, and it made the New York Times, the Wall Street Journal, it’s really gotten tremendous amount of sort of street buzz in terms of this problem. And my question is how could have the people from Facebook prevented this from happening? Or how would you prevent this from happening to your clients. That to me is an interesting question. I want to start with Nate. Nate, you’ve got this product out there, and you’re getting an established user base, what happens, what are you doing to make sure the changes that you make to the product are accepted and don’t cause revolt.
Nate: Well we probably haven’t gotten to the point where that can be an issue. Most of the time when we make change, we’re fixing something that’s broken, so people are pretty happy. You know we haven’t really made any major interface or functionality changes that would piss anybody off yet. I don’t think we’ve sort of gotten the chance to really cross that bridge. But, I mean, I think we’ve dealt with that probably more on the client side so far. You know, probably like you guys have with clients that are risking pissing off an extremely sensitive user base. Or not. Sensitive is the wrong word but loyal or expert.
Jared: Well, I mean, this is I think something that is going to happen more and more and more.
Nate: I mean, I think we’ve seen, I don’t know if this would have helped the Facebook situation but we’ve seen a lot, is that when you, you know, sort of give people a chance to, especially with this kind of user base which kind of reminds me of the eBay user base or any kind of….
Jared: Very much so, yeah.
Nate: But to the extent, I don’t know. How much did Facebook preview this stuff before they made it live?
Jared: Not at all as far as I know. They just announced it.
Nate: Right. So, I think that’s not always possible but it’s always great if you can sort of get, you know, people to play around with it before you launch it in full mode.
Lyle: Things like previews and beta, you know, small betas probably would have helped them a lot.
Nate: Yeah, yeah.
Rashmi: But it all starts with research.
Nate: That’s why eBay’s like a slave to their check out our new whatever. They always do that.
Jared: Well you know why, right?
Nate: If they don’t they just get hammered.
Jared: They get absolutely hammered.
Jared: The first time they got hammered was that they had gone for years without, well not years, maybe a year and a half with a graphic designer or anybody thinking about this stuff. It was all being done by their developers.
Jared: And they made several millions dollars in revenues, and somebody said you know, we’re just sort of amateurs at this. We should get a professional in here. So they bring in a professional and the professional looks at their screens and says. In the old days the eBay screens all had this sort of violent yellow background. The first thing the designer said was, “Okay, we’ve got to get rid of that background. We’re going to change it to white.” And so they go through, in those days they didn’t have CSS so they hand-coded everything to change it to white, and they relaunched it. That day, they get thousands of emails from their customers. Not only complaining about the change about the color. In fact, very few complained about the change in color. Almost all of them complained about the reduction in performance and the bugs that were added and all the things that had happened with this major change in the system. And all they changed was the color, but all these people complain about this stuff. So within hours they set the color back, right? They put it back to this ugly yellow. And then they didn’t know what to do. So what they did was over the next six weeks, they changed the background color one shade of yellow at a time. One per day until they got it down to white.
Rashmi: Yeah, and actually, we’ve done a bunch of work with eBay and one of things they really emphasize…we worked with them on this huge information architecture redesign and it’s been a few years so I can at least mention it. One of the things they really emphasized is how do you slowly, you know, you don’t want to do one day suddenly change everything. You really want to create a scheme and then unfold parts of it at a time. But going back to the Facebook issue, I’ve been thinking about it a lot and I’m really glad that you brought that up, Jared. So I think that it was probably going to be impossible to completely avoid such a situation because part of, if you hear about what sites like MySpace and Facebook are doing, is that they have become used to launching new features and being changing all the time and the user base kind of loves it, yeah. So on the one hand, they’ve been doing this a lot and maybe they just want some of the features are just have repercussions beyond what they could have anticipated but otherwise they are able to launch one of their, you know, to be operated to change all the time. Having said that I think this is social design, and social design requires a very different mentality than when you’re just designing for usability, say. Here do you need to consider what happens when one persons uses it, what happens when hundreds, when thousands and then when you know that I’m using the feature, and it’s just like kind of a social process, and what it changes in the social process. I think you have to explicitly consider what social process your system is enabling and how you are changing that social process.
DeWayne: I think that’s a great point because doing some user testing might have been valuable but you might have had a few people looking at the few bits of information coming back and saying yeah, this is kind of cool. But then the students that work for me, you know, they have a couple of hundred friends on their Facebook site. They love Facebook here at the university. It’s a different thing when all of a sudden you’ve got a couple hundred friends’ comments coming back at you and yours are going out to them. The whole weight of the full number of people I think might have changed it. They might have gotten some “yeah, this is kind of cool, this is kind of neat” before in just testing where when they roll it out then this would be the reaction they got.
Lyle: Because of their closed system I haven’t been able to see this, but I relate it to LinkedIn, which I am on and I’ve got a number of people I have connections to on LinkedIn. One of the things they do there is every, it seems like every few weeks I get an email that says, “Here are changes in you network.” And it will tell you, oh, “Jared Spool added a blog and this person’s job title changed” and then you can go look at their profile. It seems like LinkedIn has taken more of a throttle back approach and whereas Facebook took a, “We’re going to send you everything on a real time basis.” There’s probably a happy medium in there, but one way to kind of bring it in gradually, as Rashmi was talking about, is maybe to start with less frequently, less pure items coming at you as updates, and then let the users identify what are those things they want to see and how frequent they want to see.
Kyle: It should have started with research, don’t you think? I mean, don’t you think that initially, if they have a new idea for a feature, throw it out to the crowd and see what they think of it. Now, and I’m not sure which one of you is saying it, but we don’t know for sure, maybe when the idea is first thrown out there, ‘Oh yeah, it sounds great, ‘ but then when it’s implemented, ‘Oh, wait a minute, maybe I really don’t want this.’ You don’t know, really, even if you do get the answer in research, or what you think is the answer, but I think a mixture of all this needs to be done, not just testing or not just a slow release, a beta release. I think it has to start with research. Because there is all this ‘Wouldn’t it be cool if’ ideas that are thrown around, but no one’s bouncing them off the people who actually have to use the system.
Nate: Totally true, but the interesting thing for me of both E-bay and Facebook, clearly they need to flow stuff out before they implement it, just because the…
Jared: I bet they do from now on!
Nate: Yeah, I’m sure they will. But also there’s this point, and I think this is more applicable to E-bay now, but I think someday it’ll be applicable to Facebook, you know, it’s like when to ignore the users, and when to tolerate that first wave of negative feedback for something that’s ultimately going to be better for everybody. And that, I think, is the hardest. And that’s one thing that E-bay has just been stuck in, is being sort of chained to these previews, and not being able to just, you know, go against the vocal minority.
Jared: Josh Porter, what’s your take on this?
Josh: Well I think Rashmi makes a great point, that this is kind of a new sort of social design issue. The really ironic thing that I just read about this morning was that the actual implementation of the feature itself caused its own downfall. Because as you were looking on your Facebook profile, you could see that all the other people, all the other people in your network, were signing up for the protest petition. So that’s how that petition was signed by like 700,000 people in 24 hours, because the new feature itself propagated it so fast, so I found that really interesting. But one thing that I don’t think any of us have mentioned as well is that the privacy settings of everyone’s information actually stayed the same. So when, if you were seeing information in your news feed, that was information that was already available to you. The only difference is you had to work much less hard to see it. It’s kind of like moving from looking at HTML home pages to going to RSS. There’s just that much less effort in seeing information.
Jared: The MoBuzz TV people, said that this is like people who, somebody who stands naked in their window every morning, who gets upset because someone handed them a photograph of them standing naked in their window.
Lyle: Interesting analogy.
Josh: It’s kind of like, you know, leaving your house unlocked, and then – I leave my house unlocked all the time when I just go downtown or something. But if thousands of people were walking by my front door every day, and the information about whether or not my house was locked was that readily available, I probably wouldn’t leave it unlocked as much as I do.
Kyle: And having the ability to lock it. It’s all about control and privacy. They didn’t have the ability to not.
Josh: Yeah, what if it was an option?
DeWayne: Well, they did have the ability, but they had it turned off so you could go see it in their website. They didn’t have it turned off for the news feed.
Kyle: Right, right. That wasn’t a choice, right? When they initially launched this feature, they didn’t have the choice to actually make this not visible on the news feed.
DeWayne: I’m not sure if it was a choice, but it just defaulted to whatever you had on your news feed.
Kyle: That was the solution, wasn’t it? To give them an option to not include…?
DeWayne: That might have been. That might be.
Jared: I think they, yeah, I think they were originally using the settings you had for having one friend talk to another, but I think that there was a problem with that setting not being understood that it was specific to the feed.
DeWayne: And I think, I don’t know if this is really the leading edge or not, but I think a lot of things are cyclical and I think this is going to be one of them; this wide-open sharing that students, or young people, have on the web on sites like MySpace and Facebook. I think eventually that’s going to come back around where people are going to tighten down again. We’re seeing a little bit of that here at the university, where the students that work for me, and when I talk to other students; they’re starting to turn on more privacy filters on Facebook.
DeWayne: And filter people out. They start to think about things like, well, okay I’ve got pictures of my spring break in Cancun on there, and maybe a prospective employer might not want to see those. So, I’m going to turn off his access, or anyone except for my friends’ access to Facebook. They’re starting to think about things like that a lot more, I think. They’re still putting a lot out there, but it’s like one young man commented in the New York Times article, I believe, where when all of a sudden all this stuff coming back from the news feed really opened his eyes to how much is really out there. And I think as young people start to realize that; if not this group of young people, the next group coming up, will take a look at it and say, ‘Maybe this isn’t such a great idea to share every facet of my life this way.’
Lyle: I think it’s interesting point about a generational shift in privacy, and comfort with less privacy. It actually reminds me of a talk I heard. It was either the key not or the primary speaker at a UPA five or six years ago where he was basically saying, “Look, what if you lived in a world where you could see pretty much all information about everybody and they could see all your information?” So kind of flipping a big brother concept on its ear and saying, “What if there were no big brother? What if just everybody could see everything?” It’s kind of an interesting talk because it got you thinking about, well if you’re walking down the street and you can instantly see in the corner of your shades that the person your passing has a police record. Now, initially that might be shocking but after you pass 20 people in a given day, every day that have a record, suddenly it’s not so surprising anymore. A speeding ticket’s no big deal. But none of us today would want to wear a sticker around saying “I just got a speeding ticket.” Because it would be embarrassing. But..
Jared: I’m still trying to get rid of that “a” for adultery.
Josh: This is socially acceptable in some cases like the child molesters’ websites that you see. There was…
Jared: Megan’s law or something like that?
Lyle: Megan. Yes.
Josh: I remember hearing about, I think it was the state of Florida someone had created a Google Map mash-up where they showed where all the child molesters lived because that’s public information. There were some people, some privacy advocates who spoke up. I remember reading a couple of blog posts that just said that this is public information but making it this readily available is actually going to have a detrimental effect in this situation because these people now won’t event be able to live their lives at all. Whether or not that’s the goal or whether or not you agree with that is where you end up on the situation. So there are cases, there are other cases where people want to know things like that.
Lyle: I guess what I’m wonder is if the younger generation and the generation after them will have more comfort with privacy than previous generations and if so how does that just change things? So I think that this has actually started with first public websites and then blogs became places where people just poured out their feelings and had an online journal and then yet MySpace and Facebook and these kinds of things where it’s very personal. I think that generation that’s really high on the uptake on those kinds of services is exploring. What’s the right balance? What can I share publicly? Who really is going to care that I’m going to my friend’s party and that I drank too much last night? I think they’ll figure that out, right? And especially as it evens out across generations I think they’ll find a happy medium and who’s to say if maybe they’re just dipping their toe in the water right now or they’ve just jumped in and they need to back out a little bit. We’re not sure.
Rashmi: I have a slightly different analogy or not really analogy. I’m from India and society there is you live in a… I grew up in a small town and the concept of privacy is you live in houses where there’s a number of people and you don’t often always have your own room and people can… Even visitors might be able to see what books you are reading and things like that. So it’s a much less stronger concept of privacy and I find a lot of what’s happening on the web right now. You could go to my upcoming and you would know what I’m doing, what events I’m going to for the next two weeks. I would probably be uncomfortable with my neighbors knowing that much but if you really wanted to look at it you could find out. I find that what’s happening, I find the essential premise of a service like del.icio.us very interesting, that generally bookmarks are private but we are comfortable sharing them if it gives us back returns. I would completely agree that this generation is kind of figuring out. I think what’s bothersome all this information becomes searchable. It’s probably more problematic from their perspective that this is going to remain associated with them forever.
Jared: Yeah, Dana Boyd wrote a very interesting essay on this. She wrote, “In the tech world we have a bad tendency to view the concept of private as a single bit that is either zero or one. Either it’s exposed or not.” When company’s make the decision to make data visible in a more efficient manner there’s often panic and the term privacy is often invoked. Think back to when Deja made usenet searchable. The term is also invoked when companies provide new information to you on the data that you had previously given it. Think back to the shock over Gmail’s content-based ad delivery. Neither of these are about privacy in the bit sense but they are about privacy in a different sense. Privacy is not simply about the state of an inanimate object or a set of fife. It is about the sense of vulnerability that an individual experiences. When people feel exposed or invaded there’s a privacy issue. So she, is Dana a she or a he?
Jared: Okay, I got that right. Good. Dana’s talking in terms of people’s experience of privacy, right? So what we’re talking about here isn’t whether their privacy was violated or not it was their perception of the experience of that. So this an experience design problem in my mind. From that perspective I’m wondering how we prevent future experience design problems. I think that some of this had to do with the fact that a lot of the things we’ve given up in this internet age is the ability to control change. Back in the day if I got a new version of my word processor if there was a new version of my word processor I controlled when I moved to it. If I didn’t feel like moving to it today I didn’t have to move to it today. I could talk to 20 other people and see how they liked it before I bothered to move to it. But now we don’t control when the software changes out from under us. And the people who are designing it don’t realize that this is different, that you actually have to design not just the change itself but how the change will be embraced. And I don’t think we know about how to design for embraceable change.
Rashmi: I have a slightly different perspective. I think that the way that Facebook is managed is not bad. I think what’s more important is I think it’s impossible to avoid this kind of problems completely in an age of fast-moving changes and people trying out different features and it would not be great if every time they become afraid and stop taking risks. I wouldn’t want a situation where you did this long process of is this going to fly or not when you rolled it out. I think it’s better to know how to react to it, to release it maybe to a small group, to figure out how people are reacting and respond, kind of do damage control, like the Facebook crew did. I think it seems at the end of the week, in about a week it’s kind of taken care of and people are generally OK, and that’s fine in my perspective.
Jared: So you would recommend this as a process to your client: get on the Wall Street Journal because 700,000 people signed a petition that an idea that you had wasn’t as good as you thought it was.
Rashmi: I would not recommend it but I would not want them to completely stop taking risks because of this kind of fear.
Jared: Right. So one of the main contributions in my mind of experienced design expertise, the practice of experienced design is mitigating risks, right? That’s what we do! We recommend usability testing because it helps us produce something that’s more likely going to accommodate users. We focus on standard interaction design practices because it’s more likely to produce something that’s going to reduce the cost of development, it’s going to reduce the risk of having to re-do things and we are going to enhance satisfaction. So everything we do, I believe, is about risk reduction. Maybe I’m wrong about that, but that’s how I have always viewed it. We want to do paper prototyping at the beginning of the development cycle because it’s cheaper to fix problems that we discover at the beginning than at the end. It’s all about risk reduction, all about managing risk. That’s what we are about. So the question then is, is there a more efficient to way manage risk than to put it out there and have 700,000 people get upset? If so, where are we talking about that?
Rashmi: [laughs] I would say experienced designers would come in and they would say, “Yeah, don’t do this!” That is why I think they don’t know how to do it.
Jared: If they spoke with your client, would you have been able to predict this problem?
Rashmi: Probably not!
Jared: OK! So your practice is lacking somewhere.
Rashmi: I think that the current is we don’t know how to deal with these design problems. I would agree with that. I think the process that happened – remember that this is a group of people who, like the Facebook type users, who get upset and they all organize and then they are quite OK with it, they all disband. This group of users seems to me to be quite consumed with the way Facebook does things.
Nate: But I think what’s interesting about this group is that they probably experienced the same thing that a lot of other user groups do. They are just more willing to take time to be vocal about it.
Jared: Not only are they willing to take the time, but apparently the software promoted their being vocal about it.
Jared: That’s the ironic thing. But they are not the only ones. I was just reading today an article from Stephen Levy in Newsweek that talks about “World of Warcraft” players, and apparently about 1000 World of Warcraft players launched a protest march inside the game.
Jared:… and they all showed up as naked gnomes…
Nate: Awesome, that’s awesome!
Jared:… and the protest was to get the Blizzard, the people who run the game, to give gnomes more power.
Lyle: Only a special kind of geek could rig up that.
Nate:… I mean that’s going to become…
Jared: The 100 Naked Gnomes March.
Nate: I mean, people should be a little more geeky right now, but that kind of in-your-face protest is going to be a bit more common.
Jared: I mean, it’s fairly common now. We had an Internet site, a big law firm in Boston. The users revolted when they changed from static HTML Internet to Sharepoint. It was just an absolute revolt that they faced. I’m betting we are going to see this more and more and more. Josh Porter, you wrote a blog post a while ago about social design being more and more its own design discipline. Do you know what I’m talking about?
Josh: I think so, yes. I think Rashmi nailed it earlier. Especially in light of testing if the complexity of the situation becomes much more clear how could you ever catch ahead of time to see whether or not people would have a problem with their friends seeing what they are doing in a near-real time basis. That adds a whole order of magnitude of complexity to the testing at a time. I think Facebook’s initial reaction, I think they said “Calm down, breathe. We hear you.” Telling someone to calm down who’s upset and kind of riding a wave of anger probably wasn’t a great way to handle it. But they do seem to realize now that giving control is at least the first step to having happy customers, you know, happy users.
Jared: What I was getting is that I think there is a whole discipline that we’re now realizing we need to have which is this idea, you know, we have interaction designers, we have visual designers, and we have information designers and you know. I think social designers are going to be something that’s going to come forward, and these people will have to master the skills and experience to be able to reduce the risk of these problems going in.
Josh: Yeah, definitely, I agree, I mean the contexts that arise when you are talking about two or more people are just, you know, as complex as the human experience. Most of the testing that we’ve done in the past, and most of the people we know have done in the past, is sitting one person in front of a web application, or a website, or a desktop application and seeing if they can use it – without anyone else involved. You know, can they accomplish their personal path? Well that, as was mentioned earlier, that is not going to fly in this new realm where it matters just as much what my friends are doing whether or not I’m happy.
Nate: Totally. Yeah, we’ve been screwing around with trying to do some remote collaborative usability sessions. Well I think the problem with forced in-person collaborative use in testing is you can’t recreate the immediacy of when you’re really trying to use one of these tools with your friends.
Jared: Oh, absolutely!
Nate: But you can have people actually responding to their friends remotely, but it’s just hard to co-ordinate. But we’ve been screwing around with it.
Lyle: Well, I don’t know, it isn’t actually that much more complex than testing air traffic control systems 20 years ago, or whenever they started doing that. That’s a very complex scenario that you are trying to recreate in a test environment. You can do it. I think you can do it and I think there are also things you can do at the user research stages and also of course with previous betas to gauge the fit, you know, and that’s what we’re talking about here, how do you get the right fit? Because there is a balance between providing the really cool information that facilitates the social fabric versus, you know, something’s that actually harmful to the social fabric. But, I think where they really made a huge misstep was when they didn’t realize that they are playing with privacy. They are playing with peoples’ personal information. Just being sensitive, you know. At some point there’s a line you can go across and were they checking for ‘Have we gone across the line?’ I don’t know how you work and do you do design for a social networking site and not be mindful of the social nature of it and the privacy nature of it. Those two critical elements.
Rashmi: I wonder if you could do simulations, you know maybe, I don’t know, if sociologist do simulations trying to understand how these changes at the individual level, how they play up at the group level. That would be interesting to look at. I know you can’t…
Jared: I think there’s going to have to be a whole discipline of design and research and engineering that talks about all these things. It’s not just here, it’s recommendation systems and all sort of things. We noticed, for example, we were doing testing of both Amazon and Target. We had users in essence buying stuff from both of them and the software, the underlying software for both Amazon.com and Target.com, is the same software because Target, at least at the time we were testing, I don’t know if this is still the case, but at the time we were testing it, Target’s platform: they basically rent Amazon’s software platform. And so the way they handle reviews is identical. Yet the way reviews are done on both systems are completely different. People are very willing to write both positive and negative reviews on Amazon. But on Target it seems that people only ever write negative reviews. It is very hard to find positive reviews for any product. And you think about it, right, you go and buy a $15 alarm clock from Target and it comes and it works exactly as you expected and it’s fine. What motivation do you have to go back to review it? But if you buy an alarm clock and it sucks and it doesn’t work at all what motivation do you have to tell the world never to buy this alarm clock? But on Amazon people write positive reviews all the time. And as far as I can tell, I’ve never really understood the motivation of this, as far as I can tell, it’s A) a factor of they have so many users there’s just a number of them who are willing to do it, and B it’s this need to be part of the publishing community and to get your word out there and to be, you know. It’s much more an accepted activity on Amazon to make an effort to go back and review something positively.
Lyle: Or could it be that people view Amazon as an online community as much as an online store.
Jared: But not Target!
Lyle: Like del.icio.us where I feel like it’s worth contributing a positive review because I expect to get positive reviews from there the next time I go to shop. Whereas Target’s brand doesn’t speak that to me.
Jared: Exactly! It’s the exact same software system.
Lyle: So what I am saying is it has nothing to do with the software it has to do with the brand and the perception of what you are participating in when you shop there.
Jared: OK, and what I’m saying is: did Amazon mention that to Target when they sold it to them?
Lyle: Well, heck no, because it was all about the check and the terms.
Jared: Well, yeah, exactly. So now, so who is going to be on the social design side, and be out there saying, “Hey, if you want a recommendation system to work, don’t just necessarily copy what Amazon has done because it may not work in your case.”
Lyle: To be fair to Amazon, it could be that they didn’t have that foresight either. Maybe you have to see it play out and then it’s 20/20 vision and you could go well, this is why. We’d probably give you the same functionality and it would work.
Jared: I would never have predicted it…the only reason we noticed it was because our client at the time was interested in putting in a recommendation engine like theirs and putting in a review system like theirs. And they said, we really, really, really want it to be just like Amazon. And, you know, we were all gung ho with that until we saw what was happening over at Target. We said, wait a second, stop. That may not be the smartest thing to do. And, I think that five years from now, we’re going to have conferences, and maybe even less we’re going to have conferences with people showing up and having their dream panels, of how — this topic of social design and social engineering in that regard. I don’t want to use “social engineering” because it has other connotations. But this idea of social systems design, I don’t know maybe we’ll have to come up with a better name. Well, we’ll just ask people, we don’t have to come up with a better name.
Josh: I think part of the difficulty with what you’re talking about, Jared, is that Target, even though they were using the same underlying technologies, that says nothing about the underlying user base that they’re designing it for.
Josh: You know, they’re not bringing over the same people from Amazon. Amazon has been around so long, and they’ve had that system in place for so long, and people have been writing comments for so long, you know there are people with ten thousand comments they’ve written. And so those people are as important, if not more important, than any of the technological concerns, or the technological capabilities that underlie the site.
Jared: Those people have no life.
Kyle: You think about the product, though. If you’re talking about recommendations on books, a lot of people are more apt to say, “Ah, I just read this you have to read this.” than, you know, for something like an alarm clock. Is the product taken into account?
Lyle: Yes, you know maybe 10 years out Target’s system will be more populated, but do you think about these social networking sites that are involved in, you know in there, I would throw in things like Wikipedia, and the blogosphere, and Technorati those kinds of things, they’re all new. I mean, they didn’t evolve out of some existing thing. Or some existing brand.
Jared: To answer Kyle Pero’s question about was the product taken into account — in our study, the product was taken into account. The people were basically shopping for the same thing. So while it wasn’t exactly the same thing, and we could average across them. And see the targets products on Amazon and see that there were positive reviews. It’s really funny. There was one woman looking for an alarm clock on Amazon, and they had like six alarm clocks, and she was going through, and every one of them had negative reviews. And all she could say is, why do they sell these products? Nobody likes any of these products why do they even sell them?
Nate: Do you think it’s partially just volume, though? Jared, like over time as the volume increases it’ll even out a little bit?
Jared: Maybe. Or maybe there’s something I think that certainly their volume has something to do with it. Amazon, you know has ten times as many shoppers as Target does. That is certainly a factor, because I think there’s a critical mass notion. But I’m betting that’s not the only factor. That there’s a whole bunch of factors, and that’s why we say — just like we now know all sorts of factors that have to do with what makes a good link on a page that we didn’t know 10 or 15 or 20 years ago. I think there’s going to be a field of research or a field of engineering or discipline and people are going to get in there and there will be a demand for people with this sort of. I think this will be a big deal.
Rashmi: Jared, Amazon has been around I did my first study with that system six years ago at this point. So that’s definitely not new.
Jared: Well, it’s new compared to, let’s say, databases. It’s new compared to, you know, automobiles.
Nate: I think you could argue that there’s a fundamental difference once you hit a certain volume of these things. Because earlier on, only the most popular things on the planet get one or two reviews. You know, so it wasn’t something that was reliable or useful. But then all of a sudden there’s more people. If you look at Newegg, I mean, good Lord, there’s 300 reviews on a product sometimes.
Jared: Oh well, yeah. The reviews on Amazon well, there was the Tuscan milk one. I think I talked about that last time.
Kyle: That’s hilarious.
Jared: Which I’m looking at right now, and a gallon of Tuscan milk — it currently has 751 reviews. But, if I look at — no, I’m not going to find that — what I’m bringing up right now on my screen is the first Harry Potter book, which is Harry Potter and the Sorcerer’s Stone, and currently they have 5,238 reviews. And the most recent one was written September 3rd. So apparently there’s been nothing new to say about the book since last Tuesday.
DeWayne: I think there’s an impact that’s similar to online auctions too where, we’ve kind of skirted through this a bit, Amazon is the place you go and leave reviews for things. There are some other sites like epinions.com and places like that that just don’t usually end up with as much reviews as Amazon does. Back in the day eBay was big, Yahoo was trying to break in to the auction business, and so were a number of others, and they all were pretty much snuffed out as everybody fled to eBay. Yet there a few auction sites out there, and some have some specialized appeal but, by and large, eBay is still the 800-pound gorilla of auction sites and just like Amazon is with reviews. You said in your own studies, Jared, people instead of going straight to a site to buy something, they first go to Amazon and check the reviews there before they go back buy it.
Jared: Right, right. I’m looking at the clock, and we’re running close to our time. Couple of quick things before we stop, Kyle Pero, your PSI-CHI chapter has a meeting coming up. Want to talk about that real quick?
Kyle: Yeah, I mentioned it in the last podcast. This going to be the first AIGA/PSI-CHI joint meeting, and basically we’re taking a look at Razorfish and GSI Commerce and their user center design process. The whole purpose of getting the two groups together is to see how graphic artists and UX people can work better together. So anybody who can make it out Thursday 14th, it’s going to be held at GSI Commerce, Thursday at six o’clock we’ll be happy to have you.
Jared: Cool! Maybe you can tell us how that meeting went next time.
Jared: OK, Rashmi, do you have a Bay CHI meeting coming up?
Rashmi: Yes we do. It’s tomorrow if I’m not wrong, actually if I look on Upcoming. It’s tomorrow and we have…
Jared: You run these things, right?
Rashmi: Well we alternate. This week it’s Paul Whitmore’s turn.
Jared: Oh I see, OK.
Rashmi: It’s on my Upcoming, so I will make, yes. It’s on September 12th, it’s Mark Canter and Sarah Hunt and they are going to be talking about some of the issues we talked about, about social networking and what Sarah calls think-marketing.
Nate: Isn’t there something about can a bootstrap startup survive these days?
Rashmi: [laughs] We should do that, shouldn’t we.
Nate: I thought I read that in the description.
Rashmi: You did?
Rashmi: I guess I haven’t read the whole description.
Jared: It’s not your month. So. Anybody else got something coming up?
Lyle: Sure, I’ll make a quick plug. The Minnesota chapter of the UPA is having a meeting on Thursday 14th of September and the topic’s going to be card-sorting.
Jared: Card-sorting! So this like putting the spades on one side and the hearts on the other?
Lyle: Yes, it’s code for high-stakes Poker.
[all three laugh]
Jared: That’s it, I always end up naked in those games.
Lyle: That’s a not pretty sight. Thanks for that mental image, Jared!
Jared: OK, anybody else? Going once, going twice. OK, so I want to go round the horn here and have everybody share what they thought was the most interesting thing about the show.
DeWayne: I found the discussion between you and Rashmi about the scientific versus craft particularly interesting.
Jared: OK, Kyle Pero?
Kyle: I think our discussion when it lead to what is the criteria clients have when they are picking people. You know, do they look at it as a craft or science as well. I thought that was most interesting.
Jared: OK. Nate?
Nate Bolt: Let’s see. I think I found how often Jesus was sited as a bizarre panel member most interesting.
Jared: Well, We love Jesus! And Mohammed and Moses, so I get them all there.
Rashmi: And do you want to give the names of the 50,000 Hindu gods?
Jared: That’s right, that’s right. We will get the whole Mahabharata out there, including the person who had a bowling ball as a baby. I’m still trying to figure that one out. Never mind. The joke went dud in addition to the notion. Joshua Porter?
Josh: I’m just sorry we didn’t answer the how many users question.
Jared: We will answer it definitively next time.
Joshua: OK, Ok.
Rashmi: Does that mean steak will be in the comics next time then?
Jared: Sorry Joshua, what did you say?
Josh: I was going to say thank you all for having me here. This was my first time and I had a great time.
Jared: It’s because we just picked on you while you were gone. Rashmi?
Rashmi: I enjoyed everything, and it is such a pleasant, you know, break from work.
Jared: OK, who am I missing here?
Lyle: This is Lyle.
Jared: Lyle, yeah.
Lyle: I think the discussion about social design and all of that bodes revisiting again. And I think we’ll see more of that going forward.
Jared: Yeah, that will be interesting. I think that the conversation was really good, I really enjoyed sort of hearing, I like the idea for Kyle Pero’s panel a lot and I think we still need to work on Lyle’s panel a bit.
Lyle: Well what’s wrong with Eric Clapton?
Jared: The Clapton one I liked and I’m going to propose that for CHI right away. But I liked that and I also liked the conversation about Facebook and this whole notion about social design. You’ll find me blogging about it soon. So that being that we’re going to wrap up this particular edition of the SpoolCast, and I’m Jared Spool signing off and thank you all very much.
All: Thanks. Bye.