If you want to compare multiple user interfaces in a single study, there are two ways to match subjects to these multiple conditions:
- between subjects(Öbetween groups) Study design: Different subjects test each condition, so each subject is only exposed to a single user interface.
- Within subjects (or repeated measures)Study design: The same person tests all conditions (ie all user interfaces).
(Note that we use the word "design" here to refer to thatexperimental design, not the design of the page.)
For example, if we wanted to compare two rental car locations A and B by looking at how participants book cars at each location, our study could be designed in two different ways, both of which are perfectly legitimate:
- Regarding the topics: Each participant could try a single car rental site and only book a car there.
- Within the topics: Each participant could try the two car rental websites and book a car on each one.
Any type of user research involving more than a single test condition must determine whether it should be conducted between subjects or within subjects. However, the distinction is particularly important for quantitative studies.
Experimental design in quantitative studies
In contrast to qualitative studiesquantitative Usability-Studienthey should generate insights that can be statistically generalized to the entire user population. How data from such studies are analyzed depends on how the study was designed (i.e. the characteristics of the study).design experimentally).
The main purpose of quantitative usability studies is often to compare: a website against its competitors, two different iterations of a design, or two different user groups (e.g. experts and novices). As with any scientific experiment in which we want to uncover causal relationships, it involves a quantitative investigationtwo types of variables:
- Independent Variables, which are manipulated directly by the researcher
- dependent variables, which are measured (and which are expected to change as a result of the manipulation of the independent variables)
(If the study produces statistically significant results, we can say that a change in the independent variablecauseda change in the dependent variable.)
Let's go back to our original rental car example. If we wanted to measure which of the two locations A or B is better suited for the task of booking a car, we could chooseLocal(with the possible values orlevels— A and B) as independent variables, and time spent on the task and accuracy in booking a car can be the dependent variables. The aim of the study would be to check whether the dependent variables (time and precision) change as we vary the location, or whether they stay the same. (If they stay the same, neither side is better than the other.)
Next, to design our study, we need to answer the question of whether the study design should be between-subjects or within-subjects, ie, whether a study participant should be exposed to all the different conditions for the independent variable in question. our study (within subjects) or only on a condition (between subjects). The choice of experimental design affects the type of statistical analysis that should be used on your data.
It is possible for the design of an experiment to take place both within subjects and between subjects. For example, suppose that in the case of our rental car study, we would also be interested in how participants under the age of 30 behave compared to older participants. In this case we would have two independent variables:
- Years, with 2 levels: under 30, over 30
- Local, with 2 levels: A and B
For the study, we recruit an equal number of participants in each age group. Suppose we decide that each participant who is 30 years old or older books a rental car at location A and location B. In this case, the study is relative to the independent variable within subjects.Local(because each person sees both levels of this variable, i.e. both location A and location B). However, the study is between the subjects with regard toYears: A person can only be in one age group (under or over 30, not both). (Well, technically you could pick a group of under-30s and wait until they turn 30 to retest the sites, but this setup is highly impractical for most real-world situations.)
A few independent variables can dictate design choice.Yearsis one of them, as seen above. others areexpertise(if we want to compare experts and beginners),user type(if we want to compare different groups of users or people, for example business travelers versus holidaymakers) orGenre(provided a person cannot be of more than one gender at the same time). Outside of usability, drug trials are a common case of between-subject design: participants are exposed to only one treatment: the drug being tested or a placebo, not both. And sometimes the manipulation itself changes the state of the participant: for example, if you want to see which of the two curricula is more effective for teaching reading, you can't expose the same student to both, because once he's learned to read, he can. t you can unlearn it.
Which is better: between disciplines or within disciplines?
Unfortunately, there is no simple answer to this question. As seen above, sometimes your independent variables drive experimental design. But in many situations both designs may be possible.
- Between subjects minimizes learning and transfer between conditions.Once a person has completed a series of tasks on a car rental website, he or she is more knowledgeable about the domain than before. For example, you may know by now that car rental companies charge an additional fee for drivers under the age of 21 or what comprehensive insurance is. Knowing this will likely help you become more efficient at a second rental car location, even though that second location may be very different from the first.
With between-subject design, this knowledge transfer is not an issue: participants are never exposed to multiple levels of the same independent variable.
- Intersubject studies have shorter sessions than between subjects.A participant who tries a single rental car site will have a shorter session than one who tries two. Shorter sessions are less tiring (or boring) for users, and may also be better suited to unmoderated remote testing (especially since tools like UserZoom typically require very short session durations).
- Between-subjects experiments are easier to set up, especially when you have multiple independent variables.If the study is conducted within subjects, you must randomize your stimuli to ensure that there are no order effects. For example, in our rental car study, we need to ensure that participants don't always start with location A and then move to location B. The order of the locations should be random for each participant. This is easy with just two sites - randomly assign 50% of users to start with each site. But as the number of independent variables and the levels of an independent variable increase, randomization becomes more difficult to implement on some of the existing quantitative usability testing platforms.
- Projects within disciplines require fewer participants and are less expensive to implement.To detect a statistically significant difference between two states, you often need a good number of data points (typically more than 30) in each state. If you have a project within the subject, each participant provides a data point for each level of the independent variable. For our car rental study, 30 participants provided data points for both locations. However, if the study is between subjects, you need twice as many to get the same number of data points. That means double the costs.
- The design within the compartments minimizes random noise.Perhaps the most important benefit of within-subject designs is that they make it less likely that a real difference between your conditions will go undetected or be obscured by random noise.
Individual participants bring their own story, prior knowledge and context to the test. One may be tired after a long night, another may be bored, another may get great news just before studying and be happy. When the same participant interacts with all levels of a variable, they are affected equally. The happy person will be happy in both places, the tired person will be tired in both. However, if the study is between subjects, the lucky participant will only interact with one website, which may affect the final results. You need to make sure you have an equally lucky participant in the other group to counteract its effects.
In practice, investigators will not be able to assess such differences between participants; While gender, experience, and age may be consistent across groups, other participant-specific factors are difficult to predict or recognize.
Randomization: Essential for both types of design
Whether your experimental design is within subjects or across subjects, you need to take care of randomization, albeit in a slightly different way.
Previously, we discussed why randomization is important in in-subject designs: it counteracts possible ordering effects and minimizes transfer and learning between conditions.
For between-subject designs, you need to ensure that participants are randomly assigned to the conditions because you want to ensure that participant assignment does not affect the study results. So if a researcher decides that all participants he or she likes should interact with Site A, and then finds that Site A performs better than Site B, he or she won't know if he or she sees a real difference between the sites has determined. or if the result simply reflects your attribution (e.g. because people who feel they like you tend to return the favor and may be more patient or have a positive attitude during the test).
Even without such an obvious bias as your personal preferences, it's easy to go wrong with randomization. Suppose you conduct a study over four days, from Saturday to Tuesday. You might decide that you want the first half of your test users to start with Site A and the second half of your test users to start with Site B. This is not true randomization, however, as it is very likely that certain types of people are more likely to accept a study over the weekend and other types of people are more likely to sign up for their testing times during the week.
Conclusion
User research can take place between subjects or within subjects (or both), depending on whether each participant is exposed to a single condition or all conditions that vary within a study. Each of these types of experimental design has its own pros and cons; Intra-individual design requires fewer participants and increases the chance of discovering a real difference between their conditions; Cross-curricular designs minimize learning effects in all conditions, result in shorter sessions, and can be easier to set up and analyze.