Introduction to user feedback and testing

1. Overview

When you create a real-time 3D experience (or any other experience, product, or service), it should address defined user needs based on research with your target users.

User testing and feedback is an opportunity to “reality-check” your project — against the user personas you created during pre-production, and ideally with real people who review your designs, prototypes, and the real-time 3D experience throughout the development cycle.

2. What is user testing and feedback?

The purpose of user testing is to gather qualitative and quantitative information about how well your design and its current implementation is meeting the user needs that the experience is designed to address. In other words: testing the product as an experience.

The output of user testing is actionable feedback that you can evaluate and use to improve the experience you are creating.

User testing isn’t a once-and-done activity. User testing must be embedded throughout the phases of production. It will ensure a high-quality final product that meets the needs of your target audience. Engaging in the user feedback process will also help you develop your skills in scoping, designing, and implementing solutions that address particular user needs.

3. Why conduct structured user testing?

Although it’s possible to get valuable information from informal feedback, user testing isn’t just putting your product in front of someone and asking what they think. It’s a structured process designed to get as much specific and useful information from the tester as possible. A coordinated approach to user testing will also enable you to collect data from a range of users, so that you can compare their responses and get a better data set to support evaluation and improvement decisions.

Ideally, you should conduct several user tests at various stages of project design and development, whether you’re working on a professional project or a solo project to help you learn.

You might be tempted to think, “It’s not ready yet — I’ll show this to people and get their feedback when it’s done!” This is a natural feeling, but by the time your project is in a polished state, it’s likely you won’t be able to make big changes that the feedback might bring to light. Instead, test early and often! The more you test and refine along the way, the better your end product will be.

4. The phases of user testing

The user testing process typically follows this high level structure:

• Define the objectives.

• Plan the session.

• Facilitate the session.

• Evaluate the results.

5. Planning for user testing

Define the objectives

Before you prepare the questions to get feedback from users, you should determine:

• What you want to learn about your project
  

• What you will show the users to elicit feedback (for example, wireframes, design mock-ups, or a prototype)

You can then assess your questions and testing session plan against these objectives, to ensure that they will help you get the information you need about your project.

Prepare the questions

Next, you’ll need to prepare the questions that will guide the user’s feedback. Depending on the type of feedback you’re looking for, you might ask open-ended questions, closed-ended questions, or a combination of both.

Open-ended questions give the user freedom to explain their answers. Examples include:

• How well do you think the design/prototype/product addresses [a user need]?

• What were the moments in testing where you felt confused, and why?

• How would you improve this experience?

Although these questions can work in a survey or questionnaire, it can be more useful to ask them through a conversation. This allows you to ask follow-up questions and gain a deeper understanding of the tester’s thoughts

Closed-ended questions have predetermined answers, like those found in surveys. These might include:

• A series of statements which the user will grade their response to, for example rating confidence on a scale from 1 to 5

• Questions with simple Yes / No responses

These questions work well in a survey or questionnaire format; you don’t need to ask them yourself.

It can be helpful to use both types of questions, not only to get a variety of information, but also because some testers may find it easier to give honest critical feedback via a survey rather than to a person talking to them directly.

Plan the testing session(s)

When you’ve defined the test questions, there are logistics to consider. You should:

• Identify the best time and date.

• Determine how to find testers from your target audience.

• Identify the ideal number of testers.

• Create an agenda for the testing session(s).

• Plan to record the session (if applicable).

• Choose a leader and note-takers for the session, if working in a team.

6. Preparing yourself to run the session

User testing can be challenging to run effectively — once you’ve planned the session, you need to get ready to facilitate it. You can do this by preparing using the following tips.

Observe rather than guide

If you conduct your tests in person (in person or virtually), you may find it’s extremely difficult to sit there and not say anything while the users try your experience or review your designs. You may also want to interject and tell them how to do things, if they’re struggling! There is a balance to be struck here — if your testers are struggling with something so basic that they can’t test your product, wait at least 30 seconds, make a note of it and then tell them what to do so you can get more information. On the other hand, if they can work through the problem, their solution will be valuable information for improving your design.

Don’t explain or justify your design choices

If you gather qualitative feedback — for example, by having a conversation with the tester — again you may be tempted to explain things or justify the current implementation of your project. This can feel good, because it may “fix” the tester’s understanding — but it will not fix what you’re testing!

Don’t problem-solve during testing

The user insights that come up during testing may immediately send you into problem-solving mode, thinking about how to correct any issues raised. Your tester may also start to focus on this, exploring potential solutions they think of rather than focusing on their experience.

It isn’t that these suggestions for solutions don’t have value, but the purpose of the testing session is to gain insights about their experience and to get deep information about their issues. Prepare yourself to remain in an observational role instead, actively listening to what the tester is telling you. You can also prepare to gently steer testers back to the problem they were explaining or the next question. If you don’t feel confident doing this, try practicing with a fellow creator!

7. Facilitating the session

There are different ways a testing session might be structured, but facilitators typically use the following broad structure:

• Make introductions if testing as a group.

• Identify the goal of the session.

• Explain the recording policy (if applicable) and confirm the tester's consent to this.

• Provide the materials for testing, and observe the testers as they use or review them.

• Ask the questions you have prepared.

• Provide summaries of participant answers for clarification purposes.

• Close the session.

Tips and tricks for facilitating

Here are some useful tips and tricks to support you in facilitating user testing sessions.

• Remind yourself that you are not your product, and every product gets better with feedback — but only if you’re paying attention to what your users are telling you.
  

• Frame the experience for participants. As each tester comes in, talk with them and explain what a user test is. Emphasize that you want all feedback, and negative feedback can actually be more helpful than positive feedback. Ask them to talk or “narrate” as much as possible while using the product or reviewing the materials, so that you can get a deeper understanding of their thought processes.
  

• When the testing begins, make sure that you stop talking and record your observations in detail. Things that seem clear in the moment can easily get fuzzy if you only rely on your memory. What does the user struggle with? Is there any feature they’re not using as intended? Are they doing things that don’t seem to make sense?
  

• If the user stops talking for a while, gently remind them to turn on the interior monologue again, but otherwise pretend you’re behind a one-way mirror.
  

• If you conduct a conversation with a group of testers, make sure to include them all. Gently prompt quieter testers for their thoughts, to make sure their insights are included.

8. After the session

Immediately after the testing session, you should:

• Make sure the session was recorded (if applicable).

• Write down any additional notes or observations.

• Debrief with fellow team members (if applicable).

• Create a summary document highlighting areas to address.

• Determine what you will and will not change based on the data and feedback collected.

Evaluating and acting on feedback

When you’ve collected all the feedback, it’s time to evaluate it and think about how you will respond. Thoroughly read your notes and the questionnaires, and write down a summary of the results, whether positive or negative. This should be a bulleted list of statements, for example:

• “Users didn’t notice that there were different kinds of towers.”

• “Users found it easy to understand and use the in-context menus.”

Once you’ve done this, consider the action(s) you could take to address each issue where improvement may be necessary, and any dependencies these actions might have. This will help you evaluate the options you have for responding to the issues identified.

You don’t have to take action on every user test result, but it’s vital to listen to what your users are trying to tell you. If an important part of your product isn’t working well, it’s worth cutting some features to focus on fixing what’s there.

9. Exercise: Mini product evaluation

Before you finish this tutorial, take a quick moment to put yourself in the role of a user tester. You just need an open document or a scrap of paper, to record and organize your thoughts. To do this:

1. Choose one or two products (digital or physical) that you think are particularly well made. For example, a game or software with a particularly good user experience, or a kitchen implement that’s stylish yet functional.

2. Try to identify why you like the product so much. Use the following questions to help you:

• What user needs does it address for you?

• What is positive about the experience for you as a user?

• How do you feel when you use it?

• How steep was the learning curve when you began to use it?

• Would you recommend it to others? Extend this answer by identifying who precisely you would recommend it to, and your reason for doing so.
  

3. This is a product that you like a lot, but take a moment to identify areas it has for improvement. Use the following questions to help you:

• Do you have relevant user needs that it does not adequately address?

• Do you have desires, rather than needs, that it does not address?

• Are there areas where the experience of using it could be refined?

• If you could say one thing to the people who designed and made the product, what would it be?
  

4. Take a moment to evaluate the feedback you’ve written up for the product. If you were part of the design and development team, would this give you enough information to:

• Clearly identify the positive and negative aspects of the user’s experience of your product?

• Begin to identify possible actions that might address any user pain points?
  

If not, try to identify the additional questions you would ask to get that information — this will help you scope your questions when you prepare for user testing something that you’re working on!

Share your experience

Were you surprised by the depth of evaluation you gave on the product? Did you need to identify a lot of follow-up questions to deepen your answers, as is the case with some testers? Take a moment to share a brief reflection on the exercise and explore others’ insights in the tutorial comments!

10. Summary and next steps

In this tutorial, you’ve reviewed the process for setting up user testing, considering both the challenges it holds and the benefits it offers.

It can be hard to truly listen to what users (testers or otherwise) are telling you, particularly if the feedback is negative. After all, you’re working hard to create a digital experience — and if this is a personal project you’ve chosen, it’s likely that you have a vision for it that’s clear and compelling to you.

However, to create the best possible real-time experience, it’s important that the product can stand alone. If testers are bored by a game cutscene that you really like, then one way or another it isn’t working for the player experience. If testers say that your app controls are too complicated, it’s not because they don’t appreciate the intricate system you may have created — it’s because it doesn’t adequately meet their needs as users.

By critically evaluating the feedback and your experience, then implementing changes to better address the needs identified for your target users, you can improve the quality of your experience.

Why not try planning and running an informal testing session with fellow creators for your next portfolio project? You might be surprised at the insights you get!