The old Finnish adage “a well planned work is half completed,” also holds true for global usability testing. Planning the test takes a lot of work, but once everything has been organized, conducting the actual testing work and reporting the findings is usually relatively easy and straightforward. Consider several issues when planning a test that will be conducted in multiple locations.
Structured versus Open-ended
Deciding how highly structured the test sessions and the reporting format of the usability test will be is important. When working with multiple usability teams, it is often a good idea to make a more structured and detailed test plan than is necessary if only one or two usability specialists are involved.
The most obvious reason for this is that comparing the results between the locations can be difficult if you are not sure how the tasks were presented and what the users were required to do to complete them. This is especially true if one does not have the opportunity to attend and observe a few test sessions in each location.
A clear and well-structured test plan helps both parties when briefing the usability teams about how the test should be conducted.
We also recommend creating a well-defined report structure to avoid considerable manual editing work when compiling the results. Compiling the final report can be a tedious task if the structure of the report and the overall reporting style are not reasonably standardized.
However, making the test plan and the reporting template too structured increases the risk of losing a considerable amount of interesting and useful data. If the test or the reporting template is too strictly structured, the specialists who moderate the test sessions and write the location-specific reports cannot react to any unexpected findings that occur during the test sessions. Learning about the unexpected issues is often as important and interesting as it is to answer the predefined research questions.
Experience reveals that it is often possible to find a good balance between these issues. The key to success is to clearly define the structure and details of both the test tasks and the reporting template, while still leaving room to react to unexpected issues and novel findings, both during sessions and when reporting findings.
Setting the Goals
When creating the test plan, explicitly list the research questions to answer, based on each task. This not only helps in communicating the
goals of the test to the moderator, but also supports compiling the final report and comparing results between locations.
It is helpful to further classify the questions based on whether they deal with usability, user experience (the views and opinions of the users), or other potential issues (such as localization) relevant to the study. Explicitly labeling the research questions helps separate and define these issues both during the test sessions and when analyzing and reporting the findings.
Separating the usability and user experience issues from each other is important, since improving the usability of the product is rarely the only reason for conducting global usability tests. Most often, learning about the views, opinions, and expectations of different people in different parts of the world is at least as important as studying the usability of the product.
The same applies to listing the research questions. Don’t make the test too strictly structured, and allow for unexpected and novel findings you don’t anticipate prior to the tests.
Iterating the Test Plan
Working with multiple usability specialists provides an excellent opportunity for iterating the test plan. Fresh eyes not only help spot obvious bugs and errors in the test plan, but also provide valuable assistance in ensuring that the test can be conducted in different locations. The latter point is important, because cultural issues can affect the feasibility of the test tasks or how long it takes to complete the test.
When planning the test schedule, reserve enough time to update the test plan after gathering the feedback. Localizing the plan takes some time and performing last minute changes is not much fun anyway!
Briefing the Teams
Briefing other teams on how the test should be conducted is relatively quick and easy if you have a detailed test plan and reporting template. Most often, a two or three hour meeting, either on the phone or in person, is enough to run through the tasks and discuss what should be reported and how.
Reporting the Findings
As with briefing other usability teams, reporting the findings is a relatively straightforward exercise if one creates a good test plan and pays attention when designing the reporting template.
Decide whether you should compile the reports from the individual locations into one large report or deliver them separately. We often find that it is good practice to compile the location-specific reports into one large report that summarizes all of the findings. This not only helps comparing findings between locations, but ensures receiving a total picture of the usability and user experience of the product.
Another reason to compile into one final report is that usability problems in particular tend to be such that many of them should be addressed globally, no matter where or in how many locations they occur.
Delivering one report that summarizes all the findings does not mean that you should hide differences between locations. Quite the opposite—location specific summaries and a clear indication of where each of the findings was made are important parts of the final report. After all, the driving forces for global usability testing are the need to learn how people in different locations use the product and what they think about it. If the customer has development or marketing teams in the test locations, complement the final report by supplying them with country-specific reports too. If the reporting templates are of good quality and the reporting is done properly, this should not cause any extra work.
Summary
Moderated usability testing of a product in several countries in a short time is more a case of hard work than “rocket science.” Keys to success include setting clear goals for the study with the customer, creating a detailed test plan, and providing a good reporting template that guides presenting your findings. Careful planning and preparation ensures that the work results are what the customer expects. This also makes everybody’s work easier since the goals, and how they should be achieved, are clear. In short, you can save a considerable amount of time through good planning.
