When conducting user research, it is ideal to meet your users face to face and in the environments in which they use your product or service. For the pharmacy chain Walgreens, that means going into the stores and working with our team members inside the pharmacy. However, with a large and diverse group of users spread across all 50 states, conducting research in person is an ineffectual method for reaching large numbers of users. Walgreens created a new research initiative called research in a box (RIAB) to reach more participants while spending less money and time orchestrating on-site visits.
The Need
The demand for UX research among pharmacists and technicians in Walgreens pharmacies has steadily grown in tandem with our innovation advancements. This created a saturation of participants from around the Chicagoland area, meaning that eventually large-scale travel would be required to reach a more representative geographic sample.
So why not just use one of the many remote user testing software tools available in the market? Walgreens takes data security very seriously, and some of these third-party solutions use external sites that we block. These solutions would also require the participant to use the store’s office computer, which could interfere with the normal workflow of the store. In addition, the store computers are not equipped with video capabilities and thus don’t allow for face-to-face communication between participant and researcher.
The Plan
In order to minimize cost and time, and still gather valuable insights from our extensive network of stores, Walgreens developed the research in a box (RIAB) initiative. We used the following problem statement to guide our efforts: How can we engage our diverse, geographically distributed talent pool by using an innovative, cost-conscious, and interactive solution that allows team members across the country to easily provide feedback on concepts and designs being considered by Walgreens?
The solution was simple: Have all the equipment needed for a remote research session housed within a durable padded case and shipped directly to the stores. Once received, the pharmacy staff member unpacks the box, participates in the research session, repacks everything into the case, and ships it back to the corporate office. It sounded like an easy, cost-effective, and dynamic solution to our problem, but there were many logistical considerations. Who would receive the box and be responsible for it? How would the participant know how to set up the equipment and successfully repackage all the items? How would they ship the box back? Additionally, in order for this initiative to have any chance of success, it would require the close collaboration of the UX and business teams. In all, more than 50 team members played a role in the creation of RIAB.
After a series of cross-department meetings, workshops, and planning sessions, a basic concept structure was developed (see Figure 1) as well as a list of proposed equipment deemed essential to create the box.
Equipment:
- Hard-shell protector case
- Laptop and charger
- Mobile hotspot and charger
- Mouse
- Headset with microphone
Documents in folder:
- Equipment checklist
- Return FedEx shipping label
- FAQ document
- Extra copy of blank NDA
- Feedback survey
Then there was the question of the box itself and how we would arrange the contents inside to prevent damage during transit. A standard cardboard shipping box alone would be too flimsy and require each piece of delicate equipment to be individually wrapped in padding to protect it. This would also create more work for the team members shipping out the boxes, increasing the overall expenditures in labor and supplies each time a RIAB kit is mailed. A hard-shell protector case with a customizable foam insert was used instead. Although the upfront cost was higher, the long-term savings would start accumulating almost immediately. The customizable foam would keep all the equipment safe and make it easier for the participant to repackage everything in the correct compartment. The packaging, of course, would not replace detailed instructions for the set-up and breakdown process.
When a store is chosen to participate in an RIAB remote user research session, the selected participant is sent an email containing the following: time and date of the session, any necessary log-ons and passwords, detailed instructions for configuring the computer and WebEx, unpackaging and repackaging directions, and a Non-Disclosure Agreement (NDA) to sign. When the package arrives and the participant opens the box, they have a double-sided checklist to guide them through the set-up and breakdown processes. Upon completion of the session they need to check off items on the list to confirm that everything is accounted for (See Figures 2 and 3). A prepaid FedEx shipping label is enclosed for the participant to place on the original shipping box, reducing both cost and waste. They then bring the fully repacked box to the FedEx drop-off point within the Walgreens store. The participant completes the entire research process from box arrival to box departure without ever having to leave the store premises.
Iterative Testing and Refinement
As the true test would be if a pharmacy staff member could actually use this box to connect with a UX researcher on the other side of the country, we conducted several rounds of testing to smooth out any kinks in the process.
Pilot test
The first step was to do a proof of concept test to gauge feasibility and witness how a participant would interact with this mysterious black box of research. A team of two went to a store to deliver the box and set up video recording equipment to capture the session while a researcher in the corporate office was ready to interact with the participant. No actual research was conducted in this session; the goal was only to test the process. The researcher and business team member handed the participant the box and asked them to follow the instructions that were emailed. The observers did not intervene or assist in any way unless absolutely necessary. They timed how long various tasks took the participant to complete, such as setting up the computer, connecting to the remote session, and breaking down and repacking of all the equipment. After the session was over, the participant was interviewed about his overall experience and a user journey was created documenting delights and pain points encountered in the process (see Figure 4).
Round 2: Getting out the kinks
The main pain points uncovered from the pilot were the connectivity issues with the mobile hotspot, the VPN blockade, getting to and logging into the WebEx, and a lack of guidance on how to solve such technical problems. In response, the team created a FAQ document that participants could reference so they could troubleshoot these types of problems. We worked with the IT department to modify VPN login requirements, added a link to the WebEx on the desktop that clarified how to use the mobile hotspot so that it doesn’t accidentally shut off, and fixed the login credential typo on the instructions.
For the next round, we tested three participants new to research with our team who had varying levels of tech savviness (1 low, 1 medium, 1 high). We hoped to achieve target times of 15 minutes and under for unboxing, set up and connecting to the WebEx, and five minutes for breakdown and repacking. It took the three participants an average of 15 minutes and 30 seconds to get to the remote session started, and an average 5 minutes and 8 seconds to finish the repacking process. One user did not return the checklist to the box because “it doesn’t say to put this back in the box.” The same user also suggested having “larger, clearer titles” in the instructions and a little more “guidance”. All participants found the instructions and checklist helpful to set up and repack. One user had difficulty setting up in the beginning, but once she consulted the email instructions she easily completed the process. Additionally, there were no problems with connectivity, the VPN, or accessing the WebEx.
Round 3: The training wheels are off
After polishing up the checklist, email instructions, and FAQ document, we were ready to do a full end-to-end test. The box would be mailed to a store on the East Coast, the store would be emailed the instructions, the participant would set up the research kit at the scheduled time, connect to WebEx, engage in a full remote user research session with a researcher in Illinois, and pack up and mail the box back to the corporate office. The box arrived at the store intact and the setup and connection process took less then 10 minutes and repacking less than five minutes, according to the user. The “process was seamless” and there were no technical issues or loss of connection during the research session. The box arrived back to the corporate office a few days later with all equipment properly packed. The only issue: It was unclear if the participant or the store manager was supposed to set up the equipment. This was fixed in the next iteration of the document. With the success of this testing round we were ready to scale.
Next Steps and Future Implications
At the time of this writing, the final round of RIAB testing is wrapping up with two participants from two stores in two different states. If successful, this pilot will lead to widespread adoption and implementation of RIAB as a primary remote research program for the team. Full implementation, with up to 12 RIAB research kits out simultaneously is expected in the fall 2018. Once implemented widely, we plan to solicit user feedback on the process and integrate these insights to continuously refine and improve the methodology. To this end, we added a user feedback survey to the final research round and to all kits moving forward. We are also planning on a mobile device RIAB and tablet RIAB in the future. For these studies, we would add in a simple camera to the kit to capture participants’ behavior.
Replication
Any UX research team that would like to launch their own research in a box program will have different needs, budgets, and equipment, but there are some general recommendations that should be kept in mind. It is essential to have visual cues in instructions: Don’t just tell the user what to do; show them exactly what it looks like. Our email instructions had screenshots of what buttons to click on and what success looks like. The checklist gives a visual representation of what proper setup and breakdown looks like, which reduces user frustration and increases speed.
Invest in a strong hard-shell box that is customizable for each of your pieces of equipment. We noticed degradation of the foam insert after a few sessions, so plan to purchase more replacement inserts or start with a hard Styrofoam insert that you customize yourself. This is more work upfront, but a hard insert will degrade more slowly. Before testing with your primary users, get feedback on your documents from multiple perspectives and test the functionality of your equipment repeatedly onsite and offsite to avoid time-consuming technical difficulties.
If you want to replicate this methodology, you can begin inexpensively using equipment you already have, starting with one or two boxes and systemically expanding over time. Using RIAB allows research teams to give a voice to the greatest number of users regardless of income or internet access, leading to a wider diversity of research participants and more robust results.
Conclusion
The implementation of the research in a box initiative has been a welcome addition to the Walgreens UX research toolkit and has received broad support throughout the organization. From a financial standpoint, the cost savings are expected to be substantial. It currently costs an average of $2,200 to send two researchers to a store far enough away to require a flight. Now, they can conduct research at the same store using RIAB for just the price of round-trip shipping ($10 to $20 on average, after initial startup costs). Research in a box has allowed us to more effectively involve our users in the design process and achieve our ultimate goal of creating the best possible products for our team members and customers. We hope that it can do the same for your organization.
