The Value of User Testing – Q&A with Validately founder

A good prototyping tool must be integrated with a good user testing platform, that’s why at the beginning of 2016, you’ll have great news from Justinmind: full integration with Validately, one of the best rapid user testing platforms for prototypes and active features.

Steven Cohn, the founder of Validately, gives some practical advice on how to conduct proper user testing.

Why should a product team allocate its limited time to user testing?

Let me hit you with some industry stats:

• 50% of the features in a typical product are unused by its customers. That’s right, half of what you are designing and building right now, will not be used by customers.
• 30% of a typical development team’s efforts are classified as “re-work”. Meaning, re-coding an existing feature to better reflect customer requirements or to make it more usable.

Those are massive numbers. Real needle moving stats and it is wide spread. The truth is,

It is impossible for a designer or a product manager to sit in a room and guess how a customer wants to use a feature/product.

What are the different testing options? 

I’ll explain user testing using this grid:

When you are testing a new feature design, you want to learn 2 things:

1. UI Usability
2. Real Life Usability

UI Usability is easy to understand. Can someone go from page 1 to page 2 to page 3 to page 4 without getting confused?

Real Life Usability is more complicated. This test type answers whether a user will actually use the feature in their real life. As simple as it sounds, there is a lot that goes into that answer.

Remember this one thing when testing Real Life Usability:

What a customer says they will do, what a customer says they did and what a customer actually does are three different things.

Some example questions for when testing Real Life Usability are:

• How will they use the feature in their daily life? How frequently? In which circumstances?
• What do they view as alternatives to that feature? What do they like about that alternative? What do they hate about that alternative? What will make them switch from that alternative?

I can go on and on about the questions that help predict the success of a feature design. But the main point is this:

Until you put a prototype of a new feature in front of a customer, there is NO WAY to determine whether it meets the customer’s requirements to actually use it.

Simply gathering “requirements” in interviews is not good enough. Customers have to see something visually and try it out to tell you if they would use it.

With this basic understanding of the goals of user testing, the next part is how.

Moderated Testing

Moderated Testing means testing with a customer in real time. It can be remote or in person, but you are live with the customer asking them to try a prototype or already built feature and asking them Real Life Usability questions (noted above). You can also learn UI Usability with Moderated testing.

The benefit of Moderated testing is that you learn deep insights on how a customer will use a feature design. You can follow up and ask questions when a user mentions something and dig deep. However, this type of testing is a bigger time commitment.

This type of test is best done using a prototype because it’s easy to make changes. You’ll need to recruit testers that meet your persona & schedule a time to do a live session; run the tester through a prototype and ask them to complete specific tasks; ask Real Life Usability questions, noted above; take notes and record the session to share important aspects with stakeholders.

It would be good to speak with 5 testers per persona. If you don’t have time or budget for 5, then 3 is better than zero.

Unmoderated Testing

Unmoderated testing is asynchronous. You set up a test, send it out to people and record their interactions, and often their voices, as they attempt to complete certain tasks. You are not live with them.

The major benefit is that it’s quick because it is asynchronous. You can get large volumes of responses while you are working on something else. However, you can’t ask follow up questions and dig deeper while a user is being tested. Thus you can really only learn UI Usability. Unmoderated is NOT helpful for learning Real Life Usability.

This test is best done with a clickable prototype, because you are certainly going to learn and want to make changes. You’ll need to recruit testers that meet your persona from either a panel or on your own. You give test takers a task to complete on the clickable prototype and measure how many can complete the task within a reasonable time period.

Often you want the participant to share their thoughts out loud as they attempt to complete a task. You can ask qualitative questions afterward such as the SUS scoring approach.

Tasks should not be overly prescriptive and should not use specific words that are used in the UI. For example:

• Bad task: Go to the Dashboard and “Add A New Story” to the queue. [since there are buttons labeled Dashboard and Add A New Story in the UI]
• Good task: Add a story to your backlog

There are two approaches with unmoderated testing in terms of number of testers:

• High volume click tests: typically 100 respondents and you just look at click patterns and qualitative responses
• Talk-alouds: this is when the users voice their thoughts outloud. You only need 5 testers per persona to learn 85% of the usability issues.

How does Validately help with User Testing?

When we speak with Product Managers, UX Designers and User Researchers about why they don’t conduct more User Research/Testing, they usually say the same thing…I don’t have the time! When we delve into this this is, it’s really because the process around the actual test sessions takes longer than the testing sessions. Specifically the following pain points:

• Finding representative customers (Personas) to take tests
• Scheduling these tests
• Running these tests without fooling around with recording software
• Producing post-test reports in a shareable manner for stakeholders

Validately takes the pain out of user research/testing by automating all of these steps for our customers. We attack these pain points to make it really easy to talk to your Persona and test prototypes or live code. We offer simple and affordable solutions for both Moderated and Unmoderated testing.

How does prototyping help with user testing? 

Waiting to run user tests on coded features is an extremely inefficient and costly approach. Here’s the data (IBM):