Thursday, May 25, 2006

usability testing & soy lattes

As far as I can tell (and I've done extensive research) they have nothing in common.

However, for a startup that hasn't accepted VC funding, the thought of two-way-mirror focus group research may seem both frivolous and needlessly expensive (but mostly the latter). That's certainly what I thought. It was in a conversation on the T with Aaron that he suggested some kind of testing to observe how new users interacted with the site. The lightbulb flickered.

We're coming up on reddit's one year anniversary and we hadn't once watched -- literally watched -- someone use our site for the first time. Sure, we'd been in the presence of friends who were checking up on new links, but they'd all done it many times before.

I'm talking about watching total strangers, most of whom are frank enough to tell you what's wrong (alcohol would guarantee near-absolute candor, but at the cost of some accuracy). This is where the soy latte comes in. Incidentally, latte is Italian for bribe.

It started at Starbucks, looking for users who'd be our target market for lipstick, reddit with the cherry-flavored gloss of celebrity gossip. After that, I went to Diesel (the green mermaid's foil across the street) to seek out future redditors.

With Powerbook in hand, it was just a matter of approaching subjects with a smile and the offer of a free coffee, pastry, or anything (really, startup founders don't have the same inhibitions most people do) to get them to try out a new website. Most people turned down the coffee, of those that didn't, none ordered a soy latte.

Caffeinated or not, they sat down and diligently tinkered with the site (all the while thinking aloud as I'd instructed). What we learned was quite interesting and my only regret is that we hadn't done it sooner, but hopefully we can make good use of what we discovered.

2 comments:

masukomi said...

but.... what did you discover?

I'm sure many of they things you discovered could be useful insights to other site designers, assuming of course they're smart enough to take them in the context of the current state of reddit.

kn0thing said...

If I told you, the guys would beat me up.

Our most recent UI fix was adding a nifty new register/login layer to show new users that if they want to vote (and in the process have reddit start learning what they like) they'll need an account.

What we thought was sufficient on the old setup turned out to not be explicit enough, but we'll see how the new one does...

Sorry, I know it's a copout, but the one big lesson from this testing was that one can't expect anything from a new user. This seems rather obvious, but you'd be surprised.

To [poorly] quote a rather accomplished startup founder, "You give a user a butter knife and a minute later he's telling you it's dull and that his hand hurts."