This article is base on my final User Experience Design master project, it was my pleasure to be published on the digital magazine nosolousabilidad.com

The purpose of this article is to introduce the usability testing and the Virtual Reality (VR), and finally, define the full process to apply user testing method in VR environments. You will find each phase and steps defined, moreover requirements and recommendations about how to adapt the usability tests in VR environments, showing a case study of this technique and its results.

Introduction

What are usability tests?

Usability test or user testing is one of the main User-Centered Design techniques, which allow us to discover what usability problems has a product, thought the observations of how a group of users use it.

Combining the observation with the “Thinking aloud” protocol, in which the participant has to express verbally what he or she is thinking at each moment, this method allow us to understand why they are doing that.

Doing user testing you will learn how the users are using the product and why they are using in this way.

Thanks to the qualitative information that we find out, we can detect usability problems in the product to propose improvements and solutions that optimize its user’s experience.

What is Virtual Reality?

Virtual Reality or VR is a technological field where we can create new environments or replicate existing ones. A computer is used to create it and visualize them through a virtual reality glasses (named head-mounted-displays). The users put on the glass and visualize the environment in a very immersive way, because they can look around having a complete vision of 360 degrees. This feeling of presence is the main characteristic of VR, it generates the feeling of being in that place really.

Virtual Reality is an environment created by computer where the users can look around having a complete vision of 360 degrees

Why is necessary to adapt the user testing to Virtual Reality environments??

By one hand, VR is not a new technology, it began decades ago, but it is now when it start to expand and be accessible to most of people. It is because the evolution of the technology and the inversion of big companies, it generates more affordable prices and that social platforms being adapted to this technology. Nowadays, anyone can access to 360º videos using just their mobile and Internet connection.

By other hand, when a user testing is done on a website or mobile applications, all users know how to use a computer or an smartphone. However, it is different on VR environments. Most of users are the first time that use VR, it is a new environment, where we leave from the 2 dimensions screen to enter on a 3 dimensions space.

For this reason, it requires new interactions that they are not defined yet on any interaction standard. So that, the controls depends on the devices and/or the application. It is means, that even using the same devices his controls can be differents depends on the applications used.

However, this new reality is closer to our reality and therefore, the interaction is more natural and it is learned quickly.

In addition to that, it has a great potential because we can go inside whatever environment that could be designed by computer. It generate an enormous inmersion for the user, so that, Virtual Reality increases the learning and memory capacities.

Virtual Reality increases the learning and memory capacities

Due to this technology, the purpose of this study is define the basis and the approach to perform user testing in VR environments, that any UX designer can apply when facing a project in VR.

Virtual Reality

The term “Virtual Reality” is composed by two words that contradict themselves, according to the RAE (Royal Spanish Academy) it is their definitions:

• Reality: “Real and effective existence of something”;

• Virtual: “That has an apparent existence but it is not real”;

It is means, we are facing an environment that seems real but it is not. It has been created by computer, however, it is so immersive experience that we can get the feeling that we are in that place really.

The definition of the RAE (Royal Spanish Academy) is this:
• Virtual Reality: “Representation of images or object scenes created by a computer system, which give the sensations of its real existence”;

Virtual Reality is a new communication media. A new way to discover and learn powered by the inmersión of this technology.

The Virtual Reality is on development and evolution just now. The Gartner “Hype Cycle for Emerging Technologies” studie of July of 2017 show that.

Hype Cycle for Emerging Technologies, 2017

Virtual reality is on development and evolution just now.

Mixed Reality (MR) is a deeper concept than Virtual Reality. Milgram et al. (1994) defined “Virtual continuum” at 1994 as the range where the reality go to Virtual Reality. On this range we can found the Aumented Reality, where we can see virtual data over our reality, it is already very use on mobile applications; and Augmented Virtuality, where we can visualize real data inside of virtual realities.

Virtual continuum – Milgram et al. (1994)

Mixed Reality go throught the real environments to the virtual environments, it is included the Augmented Reality

The decision to apply the user testing on VR environments is due to the fact that, when we overcome the obstacles and barriers of this environment, it will be easier to adapt and extrapolate it to the rest of environments.

Process definition

The process about how to apply user testing on Virtual Reality environments is defined next.

It contents four phases with the specific actions to do in each one. The different actions or adaptations between common environments and VR environments are highlight with a cardboard icon .

1. Requeriments

The goal is to know the project goals, who are the current users and the potential users of the product.

It will extract specific information about the user experience in VR environment and the current status of this technology asking to experts.

Interviews / Focus group with Stakeholders

• Interviews / Focus group with Stakeholders
  • Project goal’s definition
• Ethnographic investigation
  • Contextual observation / Interview about the product / Study diary… (selection of techniques depend on the project characteristics)
  • User profiles definition
  • Interview with VR experts
    • It allows to know the current state of the technology and its evolution
  • User interview about the previous experience in VR
    • We will discover the current relation between users and VR in their daily life.
  • User’s need definition
    • Personas
    • A specific block of VR it is include

2. Planning

Define the test goal and the methodology, it include all resources necessaries.

Taking in account that more resources are necessary for VR environments, so that, you need more time for preparations.
Specifics and new metrics will be used, the surveys should be adapted to the communication method with the user.

• Test goal’s
• User profiles
  
  • The user experience with the new technologies and video games will influence the abilities and learning velocity in VR environment. The more experience quicker they will learn.
• Resources
  • Equipment
    • We need the specific equipment to execute the test in VR
    • VR Device
      • VR Glasses, headphone and controls
    • Computer that have the minimum requirements for the VR device
      • It can be different depend on the VR device used.
    • Video camera to record the user
    • Screen monitor on an accessible place
      • It allow us to observed what the user are viewing.
  • Place to do the test
    • Free space of at least 5m2
      • It should allow the free movement of the user
    • Configuration definition of the space and VR devices
      • To collocate and distribute the VR sensors.
• Test documentation
  • Previous survey
    • To know the user technological level and VR experience.
  • Consentiment form
  • Task definition
    • Scenery definition
      • Taking in account the knowledge and the current access to this technology from the user, to have the right context
    • Definition of the tasks, they should be very direct and schematic
      • It is because at the beginning the tasks should be communicate by voice and the user have to understand them and assimilate easily being on a new environment.
    • Task validation and time control on a VR environment
      • It is necessary a VR device to test and validate the tasks.
    • Take in account the discoverment time (~15 seconds per scene)
      • It is a 360 degree environment, so the user receive a lot of information to process.
  • SSQ Questionnaire
    • Simulator Sickness Questionnaire, it allows to analysis if the user is affected by some symptom as sick provocated by using the VR device.
  • Likert VR Interface questionnaire
    • It is focus on the controls, the learning and the interaction with the VR environment.
  • Post-test interview
    • It includes question related with the VR environment.
• Metric definition
  • Task success
  • Critical errors
  • Non critical errors
  • Navigation and interaction
    • It is a new environment and control for the most of the user yet.
  • Health and secondary effects
    • It is really important evaluate that user do not suffer any negative effect by using the VR device.
  • Learnability (Process evaluation)
    • It will evaluate how the users learn to control and use and how the process could be improved.
  • Help (Process evaluation)
    • It will evaluate when the user needs help about the device control and what is the best way to explain that without affect on their actions.
  • Communication (Process evaluation)
    • It will evaluate the best way to communicate with the user.

3. Test execution

Test execution to detect usability problem through the observation and qualitative feedback from users using the product.

We should have all devices ready and the communication with the users is one of the most important duties, taking in account that it is very probable that it will be the first contact with the technology and in consequence the user have not experience using VR devices.

• Equipment preparation
  • Room available with the VR device sensors configured 
  • Turning on the computer and launch the VR software
  • Turning on the VR device
  • Checking the conexion and calibration between computer and device
  • Testing and having ready the record software for the screen computer
  • Putting and having ready the video camera in a place where the camera record the user movement without enter in his space
  • Turning on the screen and check that we can visualize what the VR device show
  • Printing the test guide (It include tasks, interview and notes)
  • Printing tasks
  • Printing the consentiment form for the record

• User meeting
  • Explaining the usability test
    • IMPORTANT: Explaining that we are testing the application not the user, everything that he do will be right and all his feedback will be really important, so that, he should be thinking aloud during all the test.
    • Giving the reward if we have it
    • Doing the previous questionnaire if it was not done before
  • Explaining what VR is
    • Explaining what devices is going to use
  • Doing a basic tutorial about the controls and the app
    • Explaining basic interactions on the app: navigation and interaction
    • Explaining the physical controls of the device to do the basic actions
  • Answering any question
  • Putting the device on the user
  • Checking the user accomodation and right visualization
• Test execution
  • Communicating the tasks by voice
    • Observing and being sure that the user understand everything, repeating if it is necessary
  • Observing and detecting usability problems
    • Taking notes about the findings to analyze later
    • Detecting learnability problems using the device
      • Asking to the user what is the problem to check if it is a problem about his lack of experience controlling the device
        • Helping the user in this case, remembering him the controls and the interaction methods
    • Taking in account the adaptation period to the new environment and the control of the device, from 5 to 10 minutes usually
      • The period could be less if it is a experienced user with new technologies and video games
      • After this period, if the user has assimilated the control and basic usage of the device, all problems will be consequence of the app
    • Evaluating the user health during the test
      • Observing symptoms such as fatigue, sweating, stumbling, mistake movements caused by dizziness or dizziness or quiet of the user
        • Communicating to the user to check his status
        • If it is bad, stopping immediately the test and offer a break to the user
        • Taking notes about what is the cause of this negative effect

4. Test results

Getting all the usability problems found and classify by severity, adding several proposal to solve them.

Evaluating if the problems are due to the lack of experience of the users with the devices or due to the own producto. Anyway, try to solve through design and interaction improvements of the app.

• Creating a list with all usability problems found
• Taking in account the help given to the user to explain the control of the device during the test execution
  • Thinking about how to improve or solve it through the design of the app
• Evaluating the severity of the problem, which one is  critic problem or not
• Adding a description about when the problem was found
• Giving recommendations to solve the usability problems
  • Take in account the design for Virtual Reality environments 

Results

After a case of study, doing user testing with users of different ages and experience with new technologies and videogames, the method allows to discover usability problems to give solutions and improvements to improve the user experience.

Case of study: Usability problem detected and proposal for improvement

Usability problem

The user do not get interaction with an object because when he try to do it is out of his reach

Description

A sequence of images show the problem found. At the first one, the user try to get an object but she can not. Due to the heavy immersion of the technology, the user try to get the object with her own hand at the second image. Finally, after remember to the user that she can move around, the user go down to reach the object.

Proposal to improve

This problem can be solve showing a screen message indicating to the user that the object is far and she can not reach it, when the user try to get in the same direction where the object is, the message will be the recordatory to the user being aware of her freedom movement. Other solution should be allowing to the user get the object even when it is far away, reducing the effort that the user need to do this action.

Case study conclusions

Virtual Reality is a new technology and it required a learning and adaptation process for the user that we have to take in account during the test execution and the analysis of the results.

We can take advantage of the interactive environment to create an initial tutorial to learn about the application and the basic actions. We could use the app tutorial if it has or define some initial task to familiarize the user with the environment.

Taking advantage of the interactive environment to teach to the user about the application and the basic actions

The learnability period could change depend of the users. It depends so much of the experience of the user with the new technologies and video games especially, as it have been showed. We should take in account the evolutions of this device and the penetration in the market, this will reduce the period of learnability.

However, the user can be adapted relatively quick to this environment because is more natural to the reality than the current one on computers or smartphones, it generates a high satisfaction.

The immersion provoque that users enjoy so much the experience and they feel that the time pass quickly, even try to interact without the device controls as on the case study sample.

Conclusions

Virtual Reality is yet a new environment for the most of the users.

Therefore, it is on development just now, so in the next year we will see a big evolution and it will be more accessible in the markets. New inputs will be developed, users will be familiarized with this technology. Finally, standards will be defined for this kind of interactions. So that, we should be follow this advances daily.

Meanwhile, we try to realizate the user testing in the most independent method of devices and applications used. It required to define in a very optimal way the tasks to share it by voice.

It is more, we should be considerate that the user will required probably a short tutorial and a period to learn the use of the device, to finally, enjoy this.

Investigating about the tools to prototype on VR environments will be really interesting. To use this prototypes to apply user testing in an early stages of the project. Analytics information could very useful to apply, as tools as eye-tracking, and with that, add new information and metrics.

Taking in mind that VR environments required a adaptation and learning process to considerate on the analysis phase

VR environments required more resources and equipment, but we should not find any problem applying the adaptation to the method of this study.

As always, testing the sooner the better, and with more frequently as possible during the project development, because it required a less effort some change at the beginning of the project than the end. It is more important in VR environment because the times of development are longer. So that it will be very interesting investigate about rapid prototyping on VR environments.

Virtual Reality is a new environment, yet in development, so that we should follow all the changes and improvements, to get advantage of all knowledge that we can acquired to improve the process, taking in account also the evolution of the users with this technology, this experience will influence the ways of use.