MCAT Validity Research

Table of Contents

1. Introduction: Exploring the purpose of the MCAT Validity Study
2. Development: Putting the MCAT validity research program together
3. Focus: Areas of research
4. Objectives: Research questions to be answered
5. Timeline: Data collection and release of research results
6. Stay informed: Where information will be made available

Introduction

In April 2015, the Association of American Medical Colleges (AAMC) launched the fifth version of the Medical College Admission Test® (MCAT®). It had been nearly 24 years since the last time the MCAT exam was significantly reviewed. As a best practice, it is important to conduct periodic reviews of standardized tests, especially in those fields such as medicine with rapidly expanding bodies of knowledge and research.

After five years of research and development, the new MCAT exam was introduced in 2015. It was redesigned to test the knowledge and reasoning skills of aspiring physicians that are needed when they walk through the door on the first day of medical school. The new exam reflects the input of medical school admissions officers and medical school and undergraduate institution faculty about what is needed by entering medical students and what is taught in undergraduate colleges and universities.

Fairness is the foundation for the new MCAT exam

Fairness issues were front and center in the design and development of the new exam. The architects of the exam considered fairness issues from three lenses. These goals for the new exam included the following:

• Societal fairness: Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam.
• Procedural fairness: Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions.
• Exam fairness: MCAT scores have the same meaning and predict student performance equally well for examinees from different groups.

These lenses helped the exam architects focus on issues that they had the best chance of influencing. Decisions about the design and development of the new exam were made with societal, procedural, and exam fairness in mind.

For societal fairness, research on diversity, academic preparation and fairness will help identify better resources and outreach methods to provide access to good information about the MCAT exam. This includes developing no- and low-cost practice materials and expanding outreach to educationally- and economically-disadvantaged students and faculty at under-resourced institutions.

For procedural fairness, new score scales were designed to draw attention to the center of the scale and applicants who might otherwise be overlooked. Score reports were redesigned with score profiles to emphasize examinees' strengths and weaknesses on the exam and confidence bands to discourage overemphasis of small score differences.

For exam fairness, subjects added to the exam were limited to those taught widely. The new exam provides examinees more working time per question and includes more questions per section, so that scores provide better information about examinees' strengths and weaknesses on the exam.

Why is it important to conduct this research on the new exam?

Findings from the MCAT validity research will describe the exam's use and value in admissions decisions and allow for the development of resources for examinees and admissions committees.

1. To provide evidence about the value of the new MCAT exam in admissions decisions. MCAT scores serve an important role in medical school admissions, and it is important to provide empirical evidence about their value in admissions and their ability to predict student performance throughout all four years of the medical school curriculum.
2. To answer questions about the fairness and consequences of introducing the new MCAT exam for examinees, applicants, and medical students. When changes are made to a standardized exam, it is important to consider and evaluate fairness issues. It is important to evaluate how this new MCAT exam influenced preparation, and whether these changes introduced new barriers for educationally- and economically-disadvantaged students who aspire to become physicians.
3. To ensure the MCAT program complies with industry testing standards. This research is also essential to comply with industry testing standards that set forth the kinds of research questions that should be answered for high-stakes exams such as the MCAT exam. These standards call for evidence of technical qualities such as score precision, evidence that supports use of scores in admissions decision making, and evaluations of fairness.
4. To publicize clear, accurate, and important information about the new exam. It is our responsibility to provide clear, accurate, and important information about the new MCAT exam to all stakeholders. This includes guidance about what's tested and resources for students and their advisors to prepare for the exam as well as guidance about good ways of using scores from the new exam in admissions decision making.

Fairness is the foundation of the MCAT Validity Research Agenda

Two committees of medical school and pre-health representatives are evaluating the validity of the new exam. The committees are asking questions about the societal, procedural, and exam fairness of the new exam to identify resources, tools, data, and other strategies that will help ensure the new exam is fair.

• Societal fairness: Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam. Research on diversity, academic preparation and fairness will help identify better resources and outreach methods to provide access to good information about the MCAT exam.
• Procedural fairness: Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions. Research on admissions decision making will help the AAMC identify the resources, tools, and data to help admissions officers and their committees use MCAT scores in sound ways.
• Exam fairness: MCAT scores have the same meaning and predict student performance equally well for examinees from different groups. Research on predicting medical student performance will help admissions committees identify the range of MCAT scores associated with success and evaluate if scores from the new exam are free from predictive bias.

Development: Putting the MCAT validity research program together

Representatives from 21 medical schools and pre-health advisors from 2 undergraduate institutions serve on the two MCAT validity committees and collaborate in guiding the research.

To understand the predictive validity of the new MCAT exam, the AAMC formed two committees comprising representatives from 21 medical schools and two undergraduate pre-health advisors to oversee and guide this work. The logos from these representative institutions are shown above.

One committee oversees early research on the newly added test section on the Psychological, Social, and Biological Foundations of Behavior (PSBB) that began before the launch of the new MCAT exam. This committee is referred to as the MCAT PSBB committee. The other committee oversees the 10-year longitudinal research on the validity of the whole MCAT exam. This committee is referred to as the MCAT Validity Committee. Both committees meet regularly to review research progress, discuss findings to date, and make decisions on research plans and results dissemination.

Focus: Areas of research

Research is being conducted in three broad areas:

• Diversity, Fairness, and Academic Preparation: This area examines the demographic diversity of individuals taking the new MCAT exam compared to the old one.
• Admissions Decision Making: This area focuses on how admissions officers and their committees use the new MCAT scores, both alone and in combination with other data, in the admissions process.
• Predicting Academic Performance: This area aims to understand how well medical students' MCAT scores predict their academic performance at different stages of their undergraduate medical education.

Examples of research questions:

• Diversity of aspiring physicians: Is the demographic diversity of individuals who take the new MCAT exam the same as or greater than those who took the old MCAT exam?
• Fairness in predicting success: Will scores from the new exam predict academic performance equally well for students from different sociodemographic backgrounds?
• Increased breadth of academic preparation: Will more individuals learn about psychology, sociology, and biochemistry in preparation for the MCAT exam? Will more individuals with coursework outside of the natural sciences (e.g., ethics, social justice, humanities, behavioral and social sciences) apply to medical school?

Admissions decision making:

Research on admissions decision making aims to understand how admissions officers and their committees use the new MCAT scores alone and in combination with other data in the admissions process. Researchers are analyzing longitudinal national-level applicant and acceptance data and results from two surveys of admissions officers that ask about the use and importance of the MCAT scores in medical school admissions.

Importantly, researchers will learn if medical schools accept more diverse classes of medical students, and which medical schools weigh the new, more reliable section scores differently in ways that help them meet their mission and goals. Results are used to develop resources to ensure that admissions officers and their committees can use MCAT scores in good and balanced ways in medical school admissions.

Examples of research questions:

• Acceptance of applicants with moderate scores: Will medical schools increase the percentage of applicants with total scores in the middle of the MCAT score scale who are invited to interview and receive acceptance offers?
• Use of section scores: Will admissions committees use information about applicants' strengths and weaknesses from the new MCAT score reports to identify applicants who best fit their academic missions and goals?

Predicting academic performance:

Research on predicting academic performance aims to understand how well medical students' MCAT scores predict their academic performance at different stages of their undergraduate medical education.

Two groups of participants are included in this research. The first group consists of the national population of students who entered or will enter medical school in 2015, 2016, and 2017, matriculating to an accredited MD-granting medical school in the U.S. that uses the MCAT exam. The second group is a sample of medical students (also entering in 2015, 2016, or 2017) from 18 U.S. and Canadian medical schools diverse in terms of missions, applicant characteristics, and public/private status that volunteered to participate in this research.

Results provide evidence about how well scores on the new MCAT exam predict medical students' academic performance at different stages of their undergraduate medical education and can help admissions officers and their committees make good use of MCAT scores and other application information in admissions decisions.

Examples of research questions:

• Comparing the predictive validity of the old and new exams: Do scores from the new exam correlate with the academic performance throughout medical school as well as or better than scores from the old exam?
• Examining the predictive validity of the newest test section: Will scores from the PSBB section correlate with performance in medical school courses that call on the behavioral and social sciences better than other sections of the exam?
• Comparing the predictive validity of the new MCAT exam to other predictors: Do scores from the new exam add value to the academic information applicants already provide about themselves through applications and transcripts?

Timeline: Data collection and release of research results

The MCAT Validity Research will include multiple projects, use multiple methods, and analyze data from national and local (i.e. school-level) sources to answer its research questions.

The MCAT Validity Research will span approximately 10 years (from 2014 to 2023). The timeline for the study will support the evaluation of the new MCAT exam from its introduction to examinees in 2015, to the first use of MCAT scores in selecting students for the 2016 entering class, to the performance of these entering students, as well as students entering in 2017, from entry through graduation. The last year of the MCAT validity study, 2023, allows those students entering medical school in 2017 up to six years to graduate and complete the USMLE Step 1, Step 2 CK, and Step 2 CS licensure exams.

Data collection timeline:

Examinees: 2015: First data on examinees who took the new exam

Applicants: 2016: First data from applicants who submitted scores from the new exam

Medical Students: 2017: First data on 2016 medical students who entered with scores from the new exam

Findings about examinees, applicants, and medical students will be collected in waves.

First predictive validity data collection timeline:

The timeline below shows when the initial predictive validity findings will be available for medical students who took the new exam.

The 2016 entering class was the first to be admitted with scores from the new exam. Each year, these students' MCAT scores will be correlated with their grades, test scores, and other aspects of performance that are collected from entry through graduation.

• 2017: Year 1 coursework
• 2018: Year 2 coursework
• 2019: USMLE Step 1
• 2020: Clerkships, Graduate in 4 years
• 2021: Graduate in 5 years, USMLE Step 2-CK, USMLE Step 2-CS

Key MCAT validity findings (2017-2020):

• Findings about the academic preparation and demographic diversity of examinees who sit for the new MCAT exam
• Findings about how admissions officers are using new MCAT scores in admissions decisions
• Findings about how well scores from the new MCAT exam predict students' academic performance across the first two years of medical school

The MCAT Validity Committee will release key findings over the next 3 years.

Data about examinees' academic preparation and diversity, use of MCAT scores in admissions decisions, and the predictive validity of the new exam will be shared in multiple venues and formats. In addition to these data-based reports, the committee will share information about the overarching research agenda and goals, their work to define societal, procedural, and exam fairness that guides their research, and their work to identify the factors that may affect the predictive validity of the MCAT at schools that vary in terms of their curricula, missions and goals, applicant pools, and other characteristics.

Stay Informed: Where information will be made available

Findings from the MCAT Validity Study will be made available in a variety of different media and places including online and in-person.

• The Admissions Hub of the AAMC's website: www.aamc.org/admissions
• Through peer-reviewed publications (e.g., Academic Medicine)
• In the annual release of the publication Using MCAT Data in Medical Student Selection
• At regional and national meetings