back to overview
Focus on Methods: Does the modern remote usability test replace the classic usability test in the lab?
07/03/13 | Christina Karsten

Since the beginning of the 90s, a usability test, as a method for evaluation of a product’s user friendliness, belongs to the core of the user-centered design of software as well as of hardware. However, a test in the lab is cost-intensive and requires considerable effort. On this account, manufacturers may totally spare usability tests in special circumstances. Whereas user feedback is very important in every development process, several providers promote a more cost-effective and efficient alternative to a test in the lab: a so called remote usability test (RUT) which can be done via internet. Using special online tools, customers are even enabled to conduct a test autonomously without any usability expert. Providers of RUT promise a quality of insights which are as good as from the lab test.

In this regard, the question arises: is a classic usability test in the lab still up to date?

We would like to look at both methods separately.

2013-05_Methoden im Fokus_Usability Test vs. Remote Test_2

Usability test in the lab

During the classic usability test in the lab, 6-10 potential users try to accomplish typical tasks with the product, instructed by a moderator. All identified interaction problems are registered. With the help of the thinking-aloud-method, participants are asked to comment on their actions and to speak out all their thoughts. This qualitative data facilitates the interpretation of results. The moderator can ask further questions during and after the test and assist in the case of difficulties (but the moderator can also unconsciously influence the user’s opinion, for example by wording). A further advantage of a lab session is the possibility to avoid different disturbances like interruptions, noise, movements, multitasking, etc. However, this advantage is a disadvantage at the same time: the interaction with the product in such an artificial surrounding may tempt test persons to behave differently from their everyday life in which they would normally use the product. Questions like e.g. if the user is able to easily find back into an already started task after an interruption, are difficult to evaluate in the lab. Furthermore, only a limited number of participants can be tested in the lab.

For a classic usability test, a lab is necessary. Participants have to be recruited and incentivized. An experienced moderator is essential. Normally, each test is done individually with each participant. That is why all participants can be tested only successively. Furthermore, the feasibility of lab tests is often limited to a special period of time (e.g. daytime, not a holiday). On these grounds, the costs and time, as well as the efforts for the preparation and running of the lab tests are high.

2013-05_Methoden im Fokus_Usability Test vs. Remote Test_3

Remote-Usability-Test

Two different possibilities for a remote usability test should be distinguished – the moderated (synchronous) and the not moderated (asynchronous) test. Both types have in common: they have to be accomplished via internet.

During the moderated remote usability test (mRUT), a moderator and a user of a system/product participate in different places, but at the same time. Users sit in front of their own computers, while the moderator can sit somewhere else, i.e. in his office. Normally, only one participant can test the product at the same time. The moderator sees the display of the participant and leads him through the session, for example by phone or video connection. Different research studies showed that a mRUT produces results which are similar to the results from the lab. Moderated tests have two remarkable advantages compared to the lab test: the reachability of unique and regionally scattered target groups and testing in an environment which is more familiar to participants.

2013-05_Methoden im Fokus_Usability Test vs. Remote Test_4

The asynchronous/ unmoderated remote usability test (uRUT) takes place without a moderator. During the test time, hundreds of participants can test the product simultaneously by using their own computers or mobile devices. Participants test the product in their own surrounding, meaning in a natural environment, without any influences by a moderator. This approach can reduce the disposition to social desirable answers. All the users’ actions (i.e. answers to questions, mouse action patterns) are recorded automatically by the used remote tool. Sound and video recordings can be done additionally. It is possible to get the results after 24 hours and the feasibility is not limited to a special time period. Moreover, time zones are irrelevant. A remote test has to be installed only one time, and no personal support during the test is necessary. That’s why only unmoderated remote tests offer the advantages of cost and effort savings in contrast to lab test or mRUT.

However, an adverse effect of this kind of usability test is that there is no possibility for participants to ask questions and/or to get further instructions. As well, it is not possible to control the session: if test persons lose their focus or do not understand a task or a question, no moderator is there to help. If participants’ actions are recorded on video, it is proven difficult to remind them to speak out their thoughts. In the consequence, the number of identified usability problems in an uRUT is often smaller than in the lab. Taking into consideration these difficulties in the collection of qualitative data, the interpretation of the uRUT data is more complicated than in the lab. Furthermore, no additional measurements like eye tracking are possible in the uRUT. In the case customers conduct remote usability tests on their own and no expert is consulted for the interpretation of data, low fix costs can be relativized by the fact that false insights will be gained. In the worst case scenario, the test results lead wrong changes to the design which require compensations more costly than the realization of a lab usability test.

 

Now let’s look at our opening question: all the disadvantages disregarded, are unmoderated remote usability tests the method of choice, because of low costs and little efforts?

The answer is - it depends: the uRUT and classic usability test do not exclude each other. They rather meet different requirements.

If no strict budget restrictions exist, a usability test in the lab will provide more detailed and deeper insights about the usability of a product. In contrast to that, in projects with a very small budget, an online test can be conducted in order to identify at least big interaction problems. Online feedback can also be useful in case of small design changes for which a test in the lab would be too elaborate. On the other side, results from remote testing can be used to validate the results from the lab on the basis of a bigger sample. But it should be mentioned that both kinds of remote tests are limited to digital products like websites or software. A holistic test of a hardware device is not possible online. In addition, a fully functional or at least an online prototype is necessary for remote testing. It is difficult to test only partly functional click-dummies with content and graphic gaps without a moderator.

The conclusion: Depending on given frame conditions, different kinds of usability tests should be considered.

Table: Comparison of advantages of 3 usability tests





























































































Advantages Lab Test Remote Test
moderated (mRUT) unmoderated (uRUT)
further questions during and after the session are possible X X -
clarifications / assistance possible X X -
moderation / instructions possible X X -
disturbance are manageable X X -
low costs - - X
Low effort - - X
familiar use environment - X X
regional spread target groups easy reachable - X X
extensive results X X -
clear interpretation X X -
measurements like eye tracking possible X - -
software & hardware products testable X - -
also first click-dummys testable X X -


The alternative to lab and remote tests:

Since both methods of usability testing have their own prominent disadvantages, D-LABS prefers an intermediate solution: D-LABS’ user researchers collect qualitative and quantitative data in the field. In other words, we go to the users’ usual environments with the product, for example to the working place, if e.g. a new accounting software has to be evaluated. Doing so, it is possible to avoid some disadvantages of a lab test, like its artificial environment, without sacrificing its advantages like interviewing the users and gaining deep insights into the reasons.