How usable is your product? It is easy to answer that question with a few observations or user comments, but knowing whether a usability problem is widespread, whether one design performs better than another, or whether an experience has improved requires measurement.
What Is Quantitative Usability Testing?
Quantitative usability testing uses predefined tasks and metrics to measure how users perform. It can show how many participants complete a task, how long they take, how often they make errors, and how users rate the overall usability of the experience.
This guide explains when quantitative usability testing is useful, which metrics you can measure, how to determine an appropriate sample size, and how to interpret the results without overclaiming what the data shows.
- Quantitative usability testing measures user performance numerically, using predefined tasks and metrics such as task success, time on task, and error rate.
- Use it when you need to measure, compare, or benchmark usability, such as evaluating a redesign or tracking performance over time.
- A valid quantitative test requires more than a large sample. Tasks and success criteria should be defined in advance, testing conditions should be consistent, and the sample should be appropriate for the metric and level of precision you need. Nielsen Norman Group’s widely cited guidance of about 40 participants, for example, assumes a binary metric, a 15% margin of error, and a 95% confidence level.
- Research platforms can help meet these requirements at scale by standardizing task wording and conditions, recruiting participants, and collecting and organizing results consistently across rounds of testing.
- A usability metric should be interpreted in context. Compare results against an appropriate baseline or benchmark rather than treating a percentage as a universal pass or fail threshold. For reference, MeasuringU reports an average task completion rate of 78%, and the average SUS score is 68.
- Quantitative and qualitative usability testing answer different questions. Quantitative data helps measure what happened and how often; qualitative research can help explain why it happened.
When Should You Run a Quantitative Usability Test?
Quantitative usability testing is useful when you need to measure the extent of a usability problem, establish a baseline, compare experiences, or determine whether a design change improved user performance.
It is particularly valuable when the decision requires evidence that can be expressed numerically rather than a set of individual observations. For example, if a team is deciding whether to launch a redesigned booking flow, quantitative testing can show whether the new version improves booking completion and reduces the time users need to complete the booking.
Common research goals for quantitative usability testing
- Benchmarking the current experience. Establishing an internal baseline of how your product performs today, so future releases are measured against your own starting point.
- Tracking usability across releases. Running the same tasks after a redesign to see whether performance moved, and in which direction.
- Competitive benchmarking. Running identical tasks on your product and one or more competitors under the same conditions to identify the specific tasks where competitors outperform you, and to set targets grounded in what users can already achieve elsewhere.
- Choosing between two designs. Comparing performance on the same task across two versions when you need evidence rather than a preference.
- Comparing user groups. Identifying differences in usability performance across defined participant groups.
- Sizing a known problem. Qualitative testing surfaced an issue, and you need to know how many users it affects before it gets prioritized.
Quantitative testing can be moderated or unmoderated
Quantitative usability testing can be conducted in person, remotely with a moderator, or remotely without a moderator. The format does not determine whether a study is quantitative. The distinction comes from the study’s objective, design, sample, data collected, and analysis.
Unmoderated testing is the common choice for quantitative work, for a practical reason: the sample sizes that make quantitative results meaningful are expensive to reach when a researcher has to sit through every session. Unmoderated sessions run in parallel, so reaching 40 participants takes about as long as reaching 10.
Moderated testing still has a place in quantitative studies, particularly when tasks are complex enough that participants may need clarification, or when you want to collect spoken reasoning alongside the metrics. The trade-off is researcher time per participant, and sample size does not shrink because a moderator is present. The same sample of about 40 participants applies, each with a researcher in the session, which makes a moderated study costly to run.
What Makes a Quantitative Usability Test Valid?
A quantitative usability test produces a trustworthy number when three things are true before testing begins:
- The task has a defined endpoint. “Explore the dashboard” cannot be scored. “Find the total spend for last month” can, because there is a moment in the experience where the participant has either done the action or not.
- The success rule is written down in advance. Decide what counts as success before you see the data, including the edge cases. Does using site search instead of navigation count? Does completing the task after three wrong turns count? Scoring these ad hoc after the fact is how the same recordings produce two different success rates. It also helps to decide which result would change your design decision before the study runs, so the outcome is not argued after the fact.
- The sample is large enough for the precision you need. Small samples produce wide ranges of uncertainty, and a difference between two numbers can disappear entirely once that uncertainty is accounted for. Sample size is covered in detail in the next section.
Keep the tasks and the wording identical across every participant and every round of testing. The moment the task wording changes between releases, you are no longer comparing like with like, and the trend line you have been building loses its meaning. For the full process of planning and running a usability test, including recruitment and task writing, see our usability testing guide.
Which UX Metrics Does Quantitative Usability Testing Measure?
Quantitative usability testing typically produces five metrics. Four are behavioral, meaning they record what participants did. One is attitudinal, meaning it records how participants rated the experience. Most studies pair two or three behavioral metrics with one attitudinal measure, because behavior and perception do not always move together.
The most useful approach is to choose metrics based on the decision the study needs to inform. For example, if a team is deciding whether to simplify a product search experience, task success rate and time on task may provide more relevant evidence than a general satisfaction measure.
| Method | What it measures | How it’s usually tracked |
|---|---|---|
| Task Success Rate | Share of users who successfully complete a task | Each attempt marked pass or fail against a predefined success rule |
| Time on task | Time required to complete a task | Start and end timestamps |
| Error rate | Frequency of predefined errors | Errors recorded per task attempt |
| First-click accuracy testing | Whether the participant’s first action moves toward the correct goal | Correct first actions ÷ total attempts |
| SUS score | Perceived usability | Standardized 10-item questionnaire scored 0–100 |
Task success rate
Task success rate measures the percentage of participants who complete a predefined task according to criteria established before the study. It is one of the most fundamental usability measures because it directly captures whether users can accomplish what they came to do.
👉 How to calculate task success rate
Task success rate = (Number of successful task completions ÷ Total task attempts) × 100
Example: 31 of 40 participants complete checkout. 31 ÷ 40 × 100 = 78%
Define what counts as success before testing begins and apply the same criteria to every participant. If partial completion or different levels of success matter, track those as separate measures rather than changing the success rule after seeing the results.
Time on task
Time on task is how long participants take to complete a task, measured from the moment the task starts to the moment it ends. Shorter is usually better for routine tasks, but not always: on a content page or a product comparison, longer time can reflect engagement rather than struggle.
Decide in advance whether to report time for successful attempts only or for all attempts, since including abandoned attempts changes the meaning of the number. Task times also skew, because a few slow participants pull the average up. Reporting the median, or the geometric mean for small samples, gives a more representative figure than the simple average.
Error rate
Error rate measures how often participants make mistakes during a task, such as entering data in the wrong field, choosing the wrong option, or following a path that leads away from the goal. Define the error types you will count before testing, so two people scoring the same recording reach the same total.
👉 How to calculate error rate
Error rate = total errors ÷ total task attempts
Example: 40 participants make 22 errors across one task. 22 ÷ 40 = 0.55 errors per attempt.
Report errors per attempt rather than a raw total. A raw count grows with the sample, so 22 errors from 40 people and 22 errors from 20 people look identical when they describe very different experiences.
First-click accuracy testing
First-click accuracy testing is the percentage of participants whose first click lands on a correct path toward completing the task. It isolates one specific question: does the interface signal where to start? That makes it especially useful for navigation, labeling, and information architecture decisions, where the first choice shapes everything that follows.
First-click testing can run on a static screen or an early prototype, before any flow exists to complete. That makes it one of the cheapest ways to get quantitative evidence early in design.
System Usability Scale (SUS)
The System Usability Scale is a 10-item questionnaire that produces a single perceived-usability score from 0 to 100. Participants rate statements such as “I thought the system was easy to use” on a five-point scale after completing the tasks.
👉How SUS is scored
For odd-numbered items, subtract 1 from the response. For even-numbered items, subtract the response from 5. Add the ten results and multiply by 2.5.
A SUS score is not a percentage. A score of 70 does not mean 70% of anything, which is why it should be read against benchmarks rather than as a grade. For the full scoring method, interpretation scales, and common mistakes, see our System Usability Scale guide.
How many participants do you need for quantitative studies?
Nielsen Norman Group recommends about 40 participants for most quantitative usability studies. That figure rests on specific assumptions: a binary metric such as task success, a margin of error of roughly 15%, and a 95% confidence level. Change any of those, and the number you need changes with it.
That does not make 40 a universal rule. For example, the same NN/g guidance shows that a 20-percentage-point margin of error at a 95% confidence level requires 21 participants for a binary metric, while a 15-percentage-point margin at 90% confidence requires 28.
The practical question is therefore not simply “How many participants should I recruit?” but “How much precision do I need from this study, and what decision will the result support?“
Three situations usually call for testing with more than 40 participants:
- Comparing two designs. Detecting a difference between two versions takes more participants than measuring one, and the smaller the difference you need to detect, the larger the sample.
- Reporting by user segment. If you plan to compare new and returning users, each segment needs a sample large enough to stand on its own.
- High-stakes decisions. When a result will justify a major investment, a tighter margin of error is worth the cost.
Fewer than 40 can be reasonable when you only need a rough read, for example confirming that a task success rate is far below target, where even a wide interval sits clearly on one side of the line.
How to Interpret Quantitative Usability Results Correctly
Collecting quantitative usability data is only the first step. A result needs to be interpreted in relation to an appropriate comparison point and the uncertainty around the measurement.
For example, a task success rate of 85% tells you how many participants succeeded. It does not tell you by itself whether that is good enough, whether the design improved, or whether the difference from another version is meaningful.
| If you want to know… | Compare against… |
|---|---|
| Whether a redesign improved usability | The previous version |
| Whether one design performs better | Another version tested under comparable conditions |
| Whether usability meets a target | A predefined usability goal |
| How your product compares with similar experiences | A relevant external benchmark |
| Whether performance differs across users | The relevant participant groups |
Where possible, define the comparison and success criteria before running the study. This reduces the temptation to select the most favorable comparison after seeing the results.
What is a good task success rate?
There is no universal task success rate that defines a usable experience. Task difficulty, participant population, success criteria, and product context all affect what constitutes a reasonable result.
As a reference point, MeasuringU reports an average task completion rate of 78% across more than 1,100 usability tasks. The source also emphasizes that what counts as good depends on context.
Use a benchmark like this as a reference rather than a target that applies equally to every task. For a high-risk task, for example, a team may reasonably require a much higher success rate than it would for a low-consequence task.
What is a good SUS score?
A SUS score also needs context. In the benchmark data used by Sauro and Lewis, 68 represents the average SUS score and the midpoint of their curved grading scale.
That means 68 can serve as a useful reference point, but it should not be interpreted as a universal threshold for acceptable usability. The product being tested, participant population, study context, and comparison point all matter.
For detailed guidance on administering and interpreting the System Usability Scale, see our System Usability Scale (SUS) guide.
Common pitfalls to avoid when interpreting quantitative results
Reporting a number without its uncertainty
Every quantitative result is an estimate based on the participants you tested. A confidence interval gives you a way to represent the uncertainty around that estimate. Larger samples produce narrower intervals, and more variable data produces wider ones.
For example, if 70% of participants successfully complete a task, the true success rate in the broader population is unlikely to be exactly 70%. The confidence interval provides a range around the observed result under the assumptions of the statistical method used. Nielsen Norman Group illustrates this with a 70% success rate and a 15-percentage-point margin of error: under those assumptions, the corresponding 95% confidence interval would range from 55% to 85%. This is why reporting “70% success” without any indication of uncertainty can give a false sense of precision.
Treating small differences as meaningful
A higher metric does not automatically mean that one design is meaningfully better than another. Whether a difference is meaningful depends on the sample size, variability, confidence intervals, study design, and the decision being made.
Even a gap that looks large can fall inside the uncertainty. For example, if a redesign has an 80% task success rate compared with 70% for the current design, that difference does not by itself show that the redesign performs better. The sample size, confidence intervals, and study design determine how much confidence you can place in the difference. If the decision depends on detecting a difference between two versions, use an appropriate statistical test or collect enough data to estimate the difference with the required precision.
The same principle applies to time on task and other continuous measures. Look beyond the difference between averages and consider the uncertainty and practical importance of the change.
Across every metric, a benchmark or comparison gives you a reference point, not the answer. The strongest interpretation connects the metric to the decision the study was designed to inform. For the wider process of working through usability findings, including qualitative observations, see our guide to analyzing usability test results. If you are choosing tools to collect and analyze quantitative data, see our roundup of UX research tools to compare options.
Frequently Asked Questions
What is quantitative usability testing?
Quantitative usability testing measures user performance using numerical data collected during a structured usability study. Common measures include task success rate, time on task, error rate, first-click accuracy, and standardized usability scores such as SUS.
How many participants do you need for quantitative usability testing?
There is no single sample size that applies to every quantitative usability study. The appropriate number depends on the metric, desired precision, confidence level, and whether you need to compare groups. For a binary metric such as task success, Nielsen Norman Group describes 40 participants under specific assumptions, including a 95% confidence level and a 15-percentage-point margin of error.
Can usability testing be quantitative and qualitative?
Yes. A single usability study can collect quantitative metrics, such as task success rate and time on task, alongside qualitative observations, such as session recordings or open-ended feedback. Combining the two shows both how often a usability issue occurs and why users experience it. If participants think aloud during tasks, treat time-on-task results with caution, since speaking while working slows people down.
What is considered a good task success rate in usability testing?
There is no universal threshold for a good task success rate, because task difficulty, audience, and success criteria all affect results. As a reference point, MeasuringU reports an average task completion rate of 78% across more than 1,100 usability tasks. High-risk tasks, such as payments or medical forms, may warrant a much higher target.
What is a good SUS score benchmark for usability testing?
A SUS score of 68 is commonly used as a reference point for average perceived usability. It should not be treated as a universal pass/fail threshold because the appropriate interpretation depends on the product, participant population, and study context. For more detail, see our System Usability Scale (SUS) guide.
