01.

Overview

"You never truly know how something will be used, until you see it being used"

User testing is integral to ensure your delivery is a success. Involving the customer early allows you to create true customer centric services.

After extensive research, reading and learning, below details the program I put in place at Atom for user testing.

02.

Metrics

Each metric detailed below started with an industry set benchmark, which were derived from research from Measuring U. These were intended as a start point, giving us metrics to aim for within our testing, and once established each journey will come with new benchmarks to maintain and improve.

Success

A binary right or wrong. Did the customer achieve the desired task?

This metric is obtained after a customer has undertaken a set task. It gives an understanding on the overall success of a customer journey.

A industry average is set at 78%, therefore we use that as our benchmark to achieve.

SEQ (Single ease question)

Measured on a scale of 1-7, with 1 being difficult and 7 being easy gives you an understanding on how easy a customer finds a set task on your experience to complete.

Using an industry benchmark of 4.7 gives an aim to achieve at a minimum for each customer journey.

Confidence

Measured on a scale of 1-7, with 1 being unconfident and 7 being confident gives an understanding of how confident a customer is that they have successfully completed the task. Did the participants that were successful know they were, and did those that weren't know? Ideally the answer to both these questions should be yes, otherwise the experience isn't giving customers clear indication on completion of the set task.

Using an industry benchmark of 5.7 gives an aim to achieve at a mimimum from each customer journey.

Time

We can use this to understand how long a journey will take for a customer.

It also allows for comparison between versions of the design, giving an understanding on how it alters - ideally we should be striving for a reduction in time, as long task times can highlight an issue with a process.

SUM Score

SUM is derived by first taking your Success, Time and SEQ metrics as a percentage. The three percentages are added together before dividing by 3. This giving a percentage between 0 and 100.

Using this one overall score it can be used as a benchmark for your experience, either for a section of the experience or overall. Personally I use SUM to compare iterations of a design, allowing for one metric to show difference in performance across iterations. A benchmark established for this metric is 65%, so the desired output is 65% and above.

03.

Disaster Matrix

Allowing for correlating the captured metrics explained above to estabilsh performance enhancements.

SEQ & Success

Correlating SEQ and success, derives four quadrants to the experience.

• Good Experience (SEQ: +50% / Success +50%)
• Difficult (SEQ: –50% / Success +50%)
• Poor experience (SEQ: –50% / Success –50%)
• Disaster (SEQ: +50% / Success –50%)

Mapping these at participant level (P), gives an overview of how participants who were and weren't successful marked the task in terms of ease.

Building a picture of how each task (T) performed overall, gives a view of how different areas of the experience perform against each other.

Finally, taking the average of each task, gives you an accurate view of how your overall (O) experience performs from a success and SEQ perspective.

Based on the benchmarks identifed for each metric, the grey area outlined top right of the grid, is the goal for the individual and overall scores.

Confidence & Success

Correlating Confidence and success, derives four quadrants to the experience.

• Good Experience (Confidence: +50% / Success +50%)
• Unsure (Confidence: –50% / Success +50%)
• Poor Experience (Confidence: –50% / Success –50%)
• Disaster (Confidence: +50% / Success –50%)


Mapping these at participant level (P), gives an overview of how participants who were and weren't successful marked the task in terms of confidence. Did those that were successful know they were, and did those that weren't know?

Building a picture of how each task (T) performed overall, gives a view of how different areas of the experience perform against each other.

Finally, taking the average of each task, gives you an accurate view of how your overall (O) experience performs from a success and confidence perspective.

Based on the benchmarks identifed for each metric, the grey area outlined top right of the grid, is the goal for the individual and overall scores.

04.

Output

In a nutshell, data, data and data.

Each prototype testing using these metrics will provide:
• 15 videos of participants
• Commentary from participants
• Success score
• SEQ score
• Confidence score
• Average time taken
• Sum score
• Matrix outlining SEQ and Success
• Matrix outline confidence and success

Capturing the above for each set user test on an experience, gives you a vast understanding of how your experience is performing, what's working and what's not. Using this data to iterate and improve your experience.