There are many kinds of informal user research methods, or what Jakob Nielsen called “guerrilla HCI.” Examples are focus groups, contextual inquiries, coffee shop intercepts, and the like. These informal qualitative methods of user research have proved popular among UX practitioners for their simplicity, low cost, and reduction of the intimidation barrier. But lately a new method of informal user research is gaining popularity—drunk user testing (DUT).
Mixology: How Does DUT work?
A drunk user testing event combines the pleasantries of a social mixer with the research methods of guerrilla HCI. Attendees are served alcohol and food in a party atmosphere at a corporate space, sometimes with live music and games, and are asked to review one or more digital assets—typically a website or mobile app—in a speed-dating, quick heuristic review fashion. User researchers chit-chat and record observations while developers and product team members rub shoulders with the users first-hand. If all goes well, a great time is had by all—a mild combination of customer inebriation and insights.
Provenance: The Origins of DUT
Market research firms have served alcohol at events for decades; examples include new product idea mixers where startups pitch their concepts to potential investors and controlled tasting focus groups for the beverage industry. However, the alcohol itself was either secondary to the experience or was the subject being tested. Getting subjects tipsy was not a primary goal; although in April 2010, the International Journal of Market Research published in its Research Live blog a rather tongue-in-cheek op-ed piece titled, “Hear Me Out: Let’s Get Focus Group Participants Drunk.”
It was probably only a matter of time before somebody figured out that having intoxicated test subjects conduct heuristic reviews of digital media could be (possibly) valuable but (definitely) entertaining. That somebody seems to have been Three Sheets Research, with its YouTube postings of drunken testers attempting to navigate the Nike+ FuelBand activity tracker and Microsoft Windows 8 in the 2012-13 period. However, it’s questionable as to whether ThreeSheets is a real marketing firm or a spoof site, a concept whose origins probably lay in one (or several) bar crawls.
Then, in January 2013, Australian Will Dayble, director at the digital services firm Squareweave, published on YouTube “The User is Drunk,” a surprisingly reasoned and cogent analysis of design simplicity for interface design and tipsiness as a means to get there. “One of the beautiful things about great UI is that it’s not there, it’s like lovers who talk without talking,” Dayble proclaims after taking a big swig from a beer bottle, sounding like a combination of Jared Spool and Russell Brand (pre-rehab).
But it was March 2015 when front-end developer, whisky aficionado, and digerati globetrotter Richard Littauer created his website The User Is Drunk, and the idea of drunk user testing gained traction. Littauer’s premise was simple: “Your website should be so simple a drunk person could use it.” For a small fee, Littauer offered himself as just the right drunk person to give a heuristic review with the correct mix of inebriation and insight. Incredibly enough, more than fifty sites took Littauer up on his offer, and he began to get press from outlets like Gizmodo. In April, Hubspot’s Austin Knight wrote a blog post about Littauer’s slurred banter screen-capture session, noting that comments like, “It looks like your site is for large scale clients who are willing to spend money on buzzwords,” were brutally frank but addressed real copy deficiencies.
In August 2015, user onboarding software startup founder Jackson Noel from Appcues and Rachel Decker, then senior UX researcher at Hubspot, took inspiration from Littauer’s side gig and put a party spin on it, hosting their first “Drunk User Testing” event at Wistia on Tudor Street in Cambridge. It was basically a usability party with more than 100 guests, a photo booth, and other Boston startup-scene notables, such as Fresh Tilled Soil, HourlyNerd, and Klaviyo. Appcues continued to host drunk user testing events throughout 2017. In the summer of that year, Wayfair hosted “Testing on Tap: A Usability Testing Event,” which was co-produced by the company’s consumer insights and recruiting teams, along with the online liquor-store app Drizly. This year, Appcues took its drunk user testing event on the road, appearing at Covo in San Francisco’s Mission District.
Buzzkill: The Downside of DUT
Combining alcohol with corporate events is always a tricky proposition. Combining alcohol consumption and usability testing is perhaps even more problematic due to ethical, legal, logistic, and methodology concerns. DUT is a rather recent phenomenon, and there is little to no published accounts on its efficacy from an academic standpoint.
To date, UX researchers have typically considered intoxicated participants as antithetical to quality user research. Donna Tedesco and Fiona Tranquada in their book The Moderator’s Survival Guide: Handling Common, Tricky and Sticky Situations in User Research (2013) devote an entire section on how to handle a drunk or stoned participant. (“…have the number of onsite security handy just in case.”)
Alcohol can be fun. But drunkenness and addiction are not funny, and while there is a certain youthful tongue-in-cheekiness about an event labeled “Drunk User Testing,” we as UX practitioners need to consider the ethics of introducing intoxication as a variable into our research, and “users” as our users. If drunk user testing is OK, why not “fentanyl user testing” as well?
The legal challenges to drunk user testing generally fall under the categories of underage drinking, liability, and harassment. Few, if any of the DUT signup forms seem to mention the event is restricted to those over 21. Liability concerns for a DUT event are generally the same as with any corporate function, but with the added issue of inviting unvetted members of the public. To date there is little evidence that participants are asked to sign a liability waiver. Given that we now live in the #metoo era, encouraging “looser” behavior, as is inherent in an event like DUT, just increases the likelihood of unwanted attention, harassment, or worse. Appcues has posted a “Code of Conduct” on its Drunk User Testing website. As is typical with the tech profession, its main theme is a nebulous “Be Kind” message and looks like it was written by a marketing intern as opposed to an attorney with a more reasoned—dare I say sober—appreciation of the potential issues that may arise.
Anecdotally, the user feedback gathered at a “Drunk User Testing” event can be useful. “It’s great for discovering the little things, but it can get challenging towards the end of the evening,” said Lisa Zangerl, product designer for Drizly.
“In the end, it’s about getting more people involved in your research,” comments Rachel Decker, now the senior user researcher at ezCater. It is true; the more testing the better, and any opportunity to observe people using your product has value. It is clear that the UX teams who have participated in DUT recognize that the event doesn’t replace more formal testing methods but can enhance them.
However, Kelly Goldsmith, an assistant professor of marketing at Northwestern’s Kellogg School of Management, has done some interesting research on people’s willingness to tolerate deviant behavior in subjects who have consumed alcohol. We may perceive that the inebriated user is more direct and honest, but the real issue may be that we are more willing to tolerate their altered opinions and actions.
DUT: Does Usability + Cocktail Party Spoil the Research or Spoil the Fun?
Combining usability testing with alcohol in some sort of enjoyable event will likely continue to grow in popularity. But there are some heuristics that can be followed to make the evening safe, enjoyable, and productive for all:
- Ensure that all attendees are over 21; for example, require guests to show a valid picture ID.
- Include a small entrance fee; several DUT events to date have found this to be a good way to reasonably restrict attendees to a level of professionalism in addition to defraying event costs.
- Clear the event with your legal team and building security.
- Stick to beer and wine, not the hard stuff. For liability, hire a professional bartender to serve the drinks. Having people serve themselves alcohol is potentially problematic.
- Offer non-alcoholic drinks as well, especially lots of free water. Appcues at its San Francisco event includes a wonderful inclusive comment, “We love sober testing, too! So, if you prefer teetotal, don’t worry—we’ve got you covered!”
- Include free food—filling stuff like pizza or burritos.
- A little organization goes a long way: Rearrange the room into testing stations with enough space for lines to form; make sure the bathrooms are clearly marked, the internet works, there is space for coats, and block off no-go areas.
- Assemble a team of helpers: testers, note-takers, greeters, and possibly security. These people are “on duty” and should conduct themselves accordingly. Arrange for relief testers to take over stations at some point in the evening.
- Start off the evening with an introductory welcome speech and lay down the ground rules with regards to the testing protocol and acceptable behaviors.
- Tests should be conducted in a simple “speed-dating” format; allow five minutes per person per test to keep the lines flowing. Ask participants to do a general first-timer user task or offer general observations in a “think aloud” format.
- Have hard start and stop times; 2-3 hours should suffice.
- Arrange for a ride-sharing service like Lyft or Uber to be available to take people home at the end of the event.
Parties are fun. User research is good. Now that the spirit’s out of the bottle so to speak, it’s doubtful we could pour it back in.
