Experimental Design for the MCAT: Everything You Need to Know
Build confidence in MCAT experimental design by learning independent vs dependent variables, confounders, study types, and data interpretation strategies.
----
Part 1: Introduction to experimental design
Experimental design describes the manner in which a controlled experimental factor is subjected to a specific treatment in order to be compared with the factor that is kept constant. It is a systematic and efficient method that enables people to study the relationship between multiple factors and key responses via data collection, which eventually leads to new discoveries.
There are numerous modes of experimental design with different purposes, and it is important to understand these key differences for the MCAT exam. There are observational studies, experimental studies, and numerous ways in which studies can be designed. We will describe the major differences and aspects of each type of experimental design that you need to know.
For the MCAT, it will also be important to understand aspects of experimental design that may pose issues in validating the results of a study, such as confounding variables and different forms of bias. We’ll provide definitions of some commonly encountered forms of confounds and bias, but this list is not exhaustive.
(Suggested Reading: MCAT Psychology and Sociology: Everything You Need to Know for the MCAT)
At the end of this guide, you will also find a practice passage and standalone questions to practice applying this information to AAMC-style practice questions. Let’s get started!
----
Part 2: Study deisgn
Study design refers to the sets of methods and protocols that are used to collect and analyze data in a study. These design methods are applicable to many forms of research: observational or experimental, survey-based or methods-based.
Table 1 Types of Study Designs
| Type of study | Description | Example |
|---|---|---|
It is also important to understand several of the key terms involved in research.
A confounding variable is a variable aside from the independent variable that influences the dependent variable. For example, suppose that there is a correlation between ice cream consumption and the murder rate within a city. The confounding variable is that ice cream consumption likely peaks during the summer months when it is hot, and more people are out and about during the summer months as well, leading to increased murder rates overall. It is therefore not increased ice cream consumption that can explain the increased murder rates, but rather the confounding variable of the summer months that may explain it.
A moderating variable affects the strength of the relationship between two variables. They can include factors such as location, gender, race, ethnicity, religion, and other demographic factors. For example, consider a study that looks into whether watching a soccer game before eating results in increased food intake. The independent variable is whether they watch the sports game, and the dependent variable is how much food they intake. The moderating variable is a factor that affects how much the independent variable affects them. So suppose that some people in the study are from Europe, and others are from Canada (also assume that there is greater interest in soccer in Europe than in Canada). This means that theoretically, someone in Europe is more interested in the soccer game and would watch it more intently than someone in Canada, so the viewing of the soccer game might have a greater impact on them and how much they eat. So the variable of nationality moderates the strength of this relationship.
A mediating variable explains why two things are related. It mediates the relationship between the independent and dependent variables. For example, consider the statement: “Whenever I exercise late at night, I feel tired the next morning.” The mediating variable in this example is that the person did not sleep enough due to exercising late at night. Therefore, not sleeping enough mediates the relationship between exercise and being tired.
The placebo effect is defined as the effect produced by a placebo drug or treatment that cannot be attributed to the properties of the placebo itself and must therefore be due to the subject’s belief that the treatment “should work”. For example, the placebo used in some drug treatments is a sugar pill that should have no measurable impact on the outcome of the patient’s condition. Therefore, if the patient feels more recovered or that their condition has been alleviated due to the placebo, this would be a placebo effect that is attributed to the patient’s belief that the treatment should help them feel better.
There are many types of validity that are important to know.
External validity is the validity of generalized inferences in scientific research based on experiments. It is the degree to which the results of a study can be generalized to other situations and people.
Internal validity is the extent to which a causal conclusion based on a study can be warranted, which depends on the extent to which the study minimized systematic error or bias.
Face validity is the extent to which a test is viewed subjectively as examining the concept that it claims to measure.
Content validity is the extent to which a measure represents all aspects of a given social construct.
Construct validity is the extent to which a test measures what it claims to be measuring.
On the other hand, reliability refers to the overall consistency of a measure and whether the measure produces similar results under consistent conditions.
Statistical significance is the term that is used to indicate whether the difference observed between groups can be attributed to chance, or if it is likely the result of experimental changes. Statistical significance can be increased by increasing the sample size of the experiment.
In addition to forms of validity, there are numerous forms of bias to be aware of. Bias is a phenomenon that tends to skew the results in one direction. The halo effect is a cognitive bias in which an observer’s impression of another entity influences the observer’s thoughts and opinions about that entity’s intrinsic character or value. For example, a sharply dressed individual at a workplace might be judged to be more capable and competent than a fellow co-worker who is dressed casually, even if their actual skill and ability levels are exactly the same. Hindsight bias is the tendency after an event has occurred to see the event as having been predictable, even if there is no objective evidence for predicting it. Normalcy bias causes people to underestimate the possibility of a disaster occurring and its potential impact. Selection bias occurs when the selection of subjects for analysis is not randomized, thereby resulting in a sample that is not representative of the population intended to be analyzed. Reconstructive bias is related to memory, and the particular memory of interest may be difficult to recall when exposed to stress. Social desirability bias is the tendency of survey respondents to respond to questions in a way that may be viewed favorably by others, which may lead to over-reporting positive behaviors and under-reporting negative behaviors. Subjective validation bias, or personal validation effect, is a cognitive bias in which a person considers a piece of information to be correct if it has personal significance to them.
----
Part 3: Observational studies
Observational studies observe the behavior of participants in their natural environments. No treatment is imposed during an observational study, meaning there is no manipulation of variables.
In a case study, subjects are hand-picked for a detailed analysis of their unique situation Random selection does not occur here. Case studies are particularly useful when encountering a unique subject from whom they want to gain more detailed and specific information. An example of a case study is the story of HM, who was a man with epilepsy who had his hippocampus removed. Researchers analyzed his unique condition in order to learn more about the connection between the hippocampus and memory. Note that a case study focuses on constructing a narrative arc about a single individual or handful of individuals, and does not include aspects such as experimental controls or statistical analysis.
In surveys, a sampling of individual units from a population is studied using data collection techniques such as questionnaire construction and methods to improve the accuracy of responses. Examples of the survey include the U.S. census, which is performed every ten years, and poll exit surveys. Survey bias is a systematic error that is introduced into the sampling or testing process via selecting or encouraging one outcome over others. Response bias is the tendency of a person to answer questions on a survey misleadingly.
A correlational study attempts to determine if there is a relationship between two variables. It is important to note that correlation does not imply causation. Correlational techniques can be used in both experimental and observational studies. In an experimental study, you can associate a treatment with an outcome and infer causation from this association. However, correlations are typically used to describe observational studies because observational studies can only describe associations, but cannot infer causation.
The Hawthorne effect (also known as the observer effect) occurs when individuals change an aspect of their behavior as a result of their awareness of being observed. For example, participants may perform better on a task if they know they are being observed because they may feel social pressure to do well.
----
Part 4: Experimental studies
In experimental studies, there is a control that allows researchers to measure the change in one variable relative to another variable. There is a set of independent variables that are set by the researchers that are hypothesized to influence the dependent variable.
Controls are useful because they help researchers to validate the performance of the experimental set-up and inform them of what effects they can reasonably expect to observe. There are controls used in experimental studies known as negative controls and positive controls.
Table 2 Positive and Negative Controls
| Type of control | Treatment type | Theoretical result |
|---|---|---|
| Not expected to produce any results or difference in outcome | Experimental intervention does nothing | |
| Known to produce a difference in outcome | Accepted standard; shows what would happen if experimental intervention has same effect |
A cohort study is a subset of a longitudinal study in which subjects are chosen because they have some shared characteristic or experience within a defined period of time. The sample in a cohort study is recruited based on their exposure, and then the researchers follow up on the individuals to see if they develop an outcome. For example, individuals are recruited based on their exposure to physical activity and then are examined based on their outcome of cardiovascular disease. The main purpose of a cohort study is to estimate the risk of an outcome among a group of individuals.
In a retrospective cohort study, the outcome has already occurred for the people in the cohort at the beginning of the study. Researchers collect baseline data and look into the subjects’ records to understand their history regarding the outcome of interests. The study is termed “retrospective” because the design of the study occurs after the phenomenon of interest has already occurred to the subjects.
In a prospective cohort study, the outcome of interest has not yet occurred for the people in the cohort at the beginning of the study. For each subject, researchers collect baseline data and monitor the condition over time to see when the outcome occurs, and then analyze the data. The study is termed “prospective” because the design of the study occurs before the phenomenon of interest will occur to subjects.
Blind studies occur in order to reduce the effect that certain biases have on results, such as the placebo effect. In a single-blinded study, the participant is not told what group they are in to determine whether the independent variable (for example, a drug) impacts the outcome (for example, their health after taking the drug). In a double-blinded study, both the experimenters and the participants do not know which group the participants are in. This is to prevent observer bias and prevents the participants from figuring out which group they are in based on the experimenter’s unintentional cues.
----
Part 5: Experimental Design practice passage
In one experimental study, researchers were interested in determining whether mice with kidney cancer could be treated using a drug, referred to as Drug X. Researchers collected 100 mice with the same form of kidney cancer and split them into two groups: the negative control group and the experimental group. Mice in the negative control group were given an injection of saline that is known to have no measurable effect on their health and mice in the experimental group were injected with 200 µL of Drug X.
Researchers treated the mice at two different time points – once at the start of the study, and again two weeks into the study, as Drug X requires two doses, each two weeks apart from one another. In this study, the scientists administering the drugs were not aware of which mice belonged to which group as a form of controlling for bias. 50 of the mice were male, and 50 of the mice were female. They were split evenly into two groups, so each group had 25 males and 25 females. They were all one month old at the start of the study.
At the beginning of the study, the tumor size on the kidney was measured via an X-ray scan. The results of the study were determined by the change in the size of this tumor on the kidney via the same type of X-ray scan. The tumor was measured 3 times by 3 different experimenters, and the average of their measurements was taken. The study indicated that none of the mice that received the placebo injection exhibited any reduction in the size of their kidney tumor. In the experimental group, the tumors on 20 of the 25 females shrunk by 80%, while there was no change in tumor size and the tumors on the other 5 females. The tumors of 2 of the 25 males shrunk by 30% in the experimental group, while the other 23 males exhibited no change in tumor size.
Question 1: Because both the experimenters and the subjects were not aware of which group the subjects were in, this study is best described as a:
A) Single-blinded study
B) Double-blinded study
C) Retrospective cohort study
D) Prospective cohort study
Question 2: Suppose that the experimenter had unintentionally chosen all 100 mice from cages that were frequently exposed to UV radiation from the sun. What type of variable is this exposure to UV radiation?
A) Confounding variable
B) Controlled variable
C) Independent variable
D) Dependent variable
Question 3: Suppose that the experimenter had chosen all 100 mice carefully based on whether or not the experimenter subjectively felt that the mouse was “friendly”. What type of bias is this?
A) Social desirability bias
B) Hawthorne effect
C) Selection bias
D) Hindsight bias
Question 4: In addition to the existing two groups in the study, one of which received the Drug X and the other of which received a placebo injection, suppose that another group is added that receives a drug that is known to work against the specific kidney cancer. What would this group be considered?
A) Negative control group
B) Positive control group
C) Experimental group
D) Treatment group
Question 5: What is one way to increase the statistical significance of the results of this study?
A) Measure the tumor size once instead of three times
B) Increase the number of subjects to 1,000 mice instead of 100 mice
C) Reduce the number of subjects from 100 to 10 mice
D) Decreasing the significance level, or the probability that the researchers will reject the null hypothesis
Question 6: Suppose instead of administering Drug X to one group in this study, the researchers monitored these mice with kidney cancer over the course of two weeks to study the progression of their kidney tumors over time. What kind of study is this?
A) Correlational study
B) Cross-sectional study
C) Case-control study
D) Observational study
Answer key for passage-based questions
1. Answer choice B is correct. In a double-blinded study, both the experimenters and the participants do not know which group the participants are in. This is to prevent observer bias, especially during the point of result read-out (choice B is correct). In a single-blinded study, only the participants are unaware of which group they are placed in during an experiment, but the experimenters remain aware (choice A is incorrect). In a retrospective cohort study, the outcome has already occurred for the people in the cohort at the beginning of the study, which does not apply to this situation (choice C is incorrect). In a prospective cohort study, the outcome of interest has not yet occurred for the people in the cohort at the beginning of the study, which also does not apply to this situation (choice D is incorrect).
2. Answer choice A is correct. This is a confounding variable because UV radiation could influence the way in which cancer develops in a mouse, so the results of the study could be attributed to this UV radiation rather than any influence of Drug X or lack thereof (choice A is correct). A controlled variable is used as a constant and unchanging standard of comparison in scientific experimentation (choice B is incorrect). The independent variable is the one the experimenter controls (choice C is incorrect). The dependent variable is the variable that changes in response to the independent variable (choice D is incorrect).
3. Answer choice C is correct. Selection bias occurs when the selection of subjects for analysis is not randomized, thereby resulting in a sample that is not representative of the population intended to be analyzed – mice with a “friendly” disposition are not representative of all mice that exist (choice C is correct). Social desirability bias is the tendency of survey respondents to respond to questions in a way that may be viewed favorably by others, which may lead to over-reporting positive behaviors and under-reporting negative behaviors – this does not occur here (choice A is incorrect). The Hawthorne effect (also known as the observer effect) occurs when individuals change an aspect of their behavior as a result of their awareness of being observed – this does not occur here (choice B is incorrect). Hindsight bias is the tendency after an event has occurred to see the event as having been predictable, even if there is no objective evidence for predicting it, which also does not happen here (choice D is incorrect).
4. Answer choice B is correct. The positive control is a control group that uses a treatment that is known to produce a difference in the outcome. It sets the standard and shows what would happen if the experimental intervention has some sort of impact – in this case, the known working drug is positive control (choice B is correct). The negative control is a control group in an experiment that has a treatment that is not expected to produce any results or difference in the outcome. This result theoretically describes what would happen if the experimental intervention does nothing (choice A is incorrect). The experimental group is also known as the treatment group, which is the group given the intervention of interest (in this case, Drug X) (choices C and D are incorrect).
5. Answer choice B is correct. Statistical significance can be increased by increasing the sample size of the experiment (choice B is correct, choice C is incorrect). Making measurements only once instead of three times would reduce statistical significance (choice A is incorrect). Decreasing the significance level, or the probability that the researchers will reject the null hypothesis also decreases the statistical power of an experiment(choice D is incorrect).
6. Answer choice D is correct. This is an observational study because it observes the behavior of participants in their natural environments without any manipulation or changing of variables (choice D is correct). A correlational study attempts to determine if there is a relationship between two variables, which is not described here (choice A is incorrect). A cross-sectional study gives a “snapshot” of a population at one point in time, which is not the case here because the researchers are observing the mice over the course of two weeks (choice B is incorrect). In a case-control study, individuals with a certain condition (“cases”) are matched to similar individuals without the condition (“controls”) in order to identify the factors that may have led to their results or development of the condition, which is also not described here (choice C is incorrect).
----
Part 6: Standalone questions and answers
Question 1: In one study, researchers observed the performance of a group of college athletes over the course of their four-year long career. What type of study is this?
A) Longitudinal study
B) Correlational study
C) Retrospective cohort study
D) Single-blinded study
Question 2: Suppose that in one study, the participants are made aware that they are being observed by the researcher, and subsequently perform more poorly because they were nervous. What is this phenomenon known as?
A) Halo effect
B) Hawthorne effect
C) Normalcy bias
D) Reconstructive bias
Question 3: What type of validity measures the degree to which the results of a study can be generalized to other groups and situations?
A) Face validity
B) Internal validity
C) External validity
D) Content validity
Question 4: Which type of variable affects the strength of the relationship between two variables?
A) Moderating variable
B) Confounding variable
C) Mediating variable
D) Independent variable
Question 5: If one was interested in giving a snapshot of a population at a given time, which of the following study designs would be most appropriate?
A) Longitudinal
B) Cross-sectional
C) Case-control
D) Experimental
Answer key for passage-based questions
1. Answer choice A is correct. This is considered a longitudinal study because it involves observation over the course of a long period of time (4 years) (choice A is correct). A correlational study attempts to determine if there is a relationship between two variables, which is not occurring here (choice B is incorrect). In a retrospective cohort study, the outcome has already occurred for the people in the cohort at the beginning of the study, which does not occur here (choice C is incorrect). In a single-blinded study, the participants are unaware of which group they are placed in during an experiment (choice D is incorrect).
2. Answer choice B is correct. This is considered the Hawthorne effect because the individuals changed an aspect of their behavior as a result of their awareness of being observed (choice B is correct). The halo effect is a cognitive bias in which an observer’s impression of another entity influences the observer’s thoughts and opinions about that entity’s intrinsic character or value (choice A is incorrect). Normalcy bias causes people to underestimate the possibility of a disaster occurring and its potential impact (choice C is incorrect). Reconstructive bias is related to memory, and the particular memory of interest may be difficult to recall when exposed to stress (choice D is incorrect).
3. Answer choice C is correct. External validity is the validity of generalized inferences in scientific research based on experiments (choice C is correct). Face validity is the extent to which a test is viewed subjectively as examining the concept that it claims to measure (choice A is incorrect). Internal validity is the extent to which a causal conclusion based on a study can be warranted, which depends on the extent to which the study minimized systematic error or bias (choice B is incorrect). Content validity is the extent to which a measure represents all aspects of a given social construct (choice C is incorrect).
4. Answer choice A is correct. A moderating variable affects the strength of the relationship between two variables (choice A is correct). A confounding variable is a variable aside from the independent variable that influences the dependent variable (choice B is incorrect). A mediating variable explains why two things are related (choice C is incorrect). The independent variable is the variable whose variation does not depend on that of another (choice D is incorrect).
5. Answer choice B is correct. Cross-sectional studies give a “snapshot” of a population at one point in time (choice B is correct). Longitudinal studies are conducted over a long period of time and typically use a specific cohort of people (choice A is incorrect). In case-control studies, individuals with a certain condition (“cases”) are matched to similar individuals without the condition (“controls”) in order to identify the factors that may have led to their results or development of the condition (choice C is incorrect). An experimental study is too vague of a term to describe the specific situation in the question (choice D is incorrect).
