A between-subjects usability study gives different participants different versions of an interface; each person tests only one. Use this approach when trying one version could teach someone how to use another. If “multi-user” means testing several people at the same time, that is a separate logistics choice: concurrent sessions do not, by themselves, make a study between-subjects.
What is a between-subjects usability test?
In a between-subjects (or between-groups) design, participants are assigned to separate conditions. For example, one group tests the current interface and another tests a revision. Each participant encounters only one version. In a within-subjects (or repeated-measures) design, the same participants try multiple versions.
As an Amazon Associate I earn from qualifying purchases.
The distinction matters because a participant’s experience in one condition can affect what they do in another. The study design also affects recruitment needs and how results should be analyzed. Nielsen Norman Group explains the trade-offs between the two designs.
Recommended Free Tools
Should I use between-subjects or within-subjects?
Choose based on the decision the study needs to support, not on a rule that one design is always better.
#1 Best Overall
- Used Book in Good Condition
| Consideration | Between-subjects | Within-subjects |
|---|---|---|
| Learning between versions | Useful when trying one version could teach participants skills or information that would affect a later version. | Exposure to one condition can influence performance in another; condition order may also affect results. |
| Participants and recruiting | Requires separate groups, so recruiting can be more demanding. | Can use fewer participants because the same people encounter multiple conditions. |
| Participant-to-participant variation | Group differences may influence outcomes as well as the interface condition. | Comparisons are less affected by random differences between participants, though exposure and order remain concerns. |
| Analysis | Analyze outcomes as a comparison between separate groups. | Analyze repeated observations from the same participants; account for condition exposure and order. |
If you use a within-subjects design, plan for order effects and learning. Counterbalancing may be appropriate, but the exact approach depends on the conditions and study protocol.
How many users do I need to test two designs?
There is no universal participant count that guarantees reliable results. The right number depends on whether you are looking for usability problems or making a quantitative comparison, as well as on the conditions, user groups, outcomes, and desired precision.
- Qualitative issue-finding: Nielsen Norman Group’s Jakob Nielsen recommends five participants as a useful default for a qualitative usability study, with exceptions. This is not a promise that five people will reveal every problem, nor does it mean five participants are enough in each condition of a comparison. See Nielsen’s explanation of the five-user recommendation.
- Quantitative comparison: Nielsen’s 2012 guidance says at least 20 participants may be needed to obtain statistically significant numbers, and that tighter confidence intervals require more. Treat this as general guidance, not a substitute for a study-specific power or precision calculation.
- Multiple user groups: A Nielsen Norman Group planning checklist suggests 2–5 participants per group depending on participants’ experience and how much their attitudes overlap. This is planning guidance for distinct target groups, not a universal sample size for comparing designs. See the usability study planning checklist.
Do not apply the five-participant qualitative rule of thumb to a statistical comparison. Small qualitative samples can help identify breakdowns, but they do not estimate population-level success rates. For quantitative conclusions, calculate the sample needed for the actual outcome and comparison.
How to plan a between-subjects comparison
- State the decision and outcome. Decide what the study must help you choose, then define what evidence would answer that question. Exploratory testing to find usability problems has a different purpose from a quantitative comparison of versions.
- Define each condition. Specify exactly which interface or version each group will test. Use comparable tasks and session procedures so the condition is the meaningful difference.
- Recruit the intended users. Set eligibility criteria for the product’s target population. If the product serves distinct user groups, plan for each group rather than assuming one overall participant total covers every segment.
- Keep the comparison consistent. Match task wording, moderator behavior, device and context, and the level of assistance. Choose outcomes—such as task completion, errors, time, or observed breakdowns—that answer the study question.
- Assign and document participants. Record which condition each participant tested and how participants were allocated. Do not imply random assignment unless the protocol actually used it.
- Analyze according to the design. Compare the groups on the selected outcomes. In a between-subjects study, an apparent difference may reflect participant composition as well as interface condition, so describe that limitation when interpreting results.
A Nielsen Norman Group example used separate participants for each version to prevent skills learned during earlier tests from influencing later tests. It involved 10 people per design version, but that was an illustrative study—not a general sample-size recommendation. See the example of parallel design and testing.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Can I test multiple users at the same time?
Yes, but simultaneous testing is about session logistics, not the between- versus within-subjects distinction. Several participants can each test one version independently, or a study can use another design; the important question is whether each person’s behavior can be observed reliably.
For evidence about individual task behavior, preserve independent observation. Nielsen Norman Group recommends starting with one-on-one live-interface sessions before moving to group discussion in its account of multiple-user simultaneous testing. Digital.gov also notes that simultaneous independent testing followed by discussion requires enough note-takers to observe each participant; consult its usability testing guidance.
Rank #4
- Used Book in Good Condition
A group discussion can surface reactions and opinions, but it is not a substitute for observing each person attempt the tasks. If participants share a session, plan who will observe each one and keep individual findings distinct from discussion feedback.
What can the results establish?
A well-matched comparison can show how participants in the tested groups performed on the selected tasks under the study conditions. It does not automatically establish that the interface caused every observed difference: separate groups can differ in experience, confidence, or other relevant characteristics. Report the allocation method, participant mix, tasks, outcomes, and limitations so readers can judge the comparison.
For exploratory studies, report the usability issues and observed breakdowns rather than presenting a small sample’s task-success percentage as a population estimate. For quantitative claims, use a sample and analysis suited to the outcome and the precision the decision requires.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
