Beckman Foundation Announces Publication of Results from Multi-year Study on Impact of Blinded Application Reviews on Institutional Prestige Bias in eLife
Beckman Young Investigator Award Applicants Gain Improved Equity and Inclusion through Introduction of Blinded Technical Proposal Requirement
- 2,291 applications were reviewed over an eight-year period, with the blinding requirement introduced at the midpoint in 2020.
- Distribution of applicants invited to the full application phase shifted from “prestigious institutions” to other institutions outside the group with the new policy.
- Trending shift carried through to final program awards.
- Pre-blinding, 75 percent of BYI awards went to applicants from Top 25 institutions.
- After blinding, 45 percent of BYI awards went to applicants from Top 25 institutions.
- Reviewers reported blinding facilitated streamlined reviews and discussions, reduced workload, and decreased potential for burnout.
The Arnold and Mabel Beckman Foundation announced today that eLife published results from a multi-year study of the Beckman Young Investigator program which demonstrates the positive impact of blinded application reviews on mitigating institutional prestige bias. In the eight-year study of 2,291 program applicants, the instruction to blind the technical proposal in the initial Letter of Intent (name, gender, gender-identifying pronouns, and institutional information omitted) resulted in a 30 percent reduction in awards issued to applicants from Top 25 institutions. With respect to gender, no evidence of bias in applicant distribution of invites to submit full applications nor eventual award distribution was found.
“I feel strongly that Dr. and Mrs. Beckman would be extremely proud of this work by the Foundation. The mission they set us out to fulfill is an important one – we are tasked with supporting the most innovative and ambitious young scientists in the chemical and life sciences. Clearly, a biased review process would be a major hindrance in our work and these studies have shown a path forward to creating more equitable, inclusive, and effective proposal review methods.
The notable study, conducted to assess the fairness and inclusiveness of the BYI awardee selection process with respect to gender and institutional affiliation, challenges the mixed results of prior studies regarding the effectiveness of blinding or masking information for reducing reviewer bias toward certain population groups. In it, four program award cycles between 2017 and 2020 whose applicants supplied unblinded technical proposals during the letter of intent stage were compared with four program cycles between 2021 and 2024 where applicants submitted blinded technical proposals during the same stage. Other eligibility criteria for the applicants and the LOI review process itself remained essentially unchanged during the study’s evaluation period. The Foundation hypothesized that if the proposal review process was free from institutional prestige and gender biases, then there would be no difference in the distribution of LOIs which advanced to the full proposal invite stage, and the resulting final program awards, between the unblinded and blinded groups. The data yielded no evidence of bias either pre- or post-blinding with respect to gender distribution, however there were notable differences after blinding with respect to the institutional affiliation of the applicants.
One of the early steps was to define “institutional prestige” for the purposes of evaluating systemic bias within the review process. To accomplish this, the study’s authors developed institutional ranking schema divided into categories based on the Foundation’s historical funding trends as well as published institutional rankings from four independent organizations. The authors then calculated the “Relative Advantage – Full Application” by category as the ratio of each category’s percentage of submitted LOIs that went on to receive a full application invite to the percentage of total LOIs offered a full application invite.
A striking variance was exhibited between the relative advantage of the “Top 1-10 institutions” group and the “other institutions” group. Under the unblinded process, the relative advantage for the “Top 1-10” group was approximately 2.3 times that of the “other” group (Average for “Top 1-10” group = 1.6, Average for “other” group = 0.7). With blinding, the disparity in the relative advantage between these two groups was reduced, with the relative advantage for the “Top 1-10” group approximately 1.4 times that of the “other” group (Average for “Top 1-10” group = 1.2, Average for “other” group = 0.85).
“I have long suspected that young investigators at top ranked universities have a better chance of obtaining grants from foundations than their counterparts at lower ranked institutions. This well documented study by Hultgren, Patras, and Hicks shows that this sort of bias likely is widespread in the science funding space. In my view, the study will be of great value to foundation directors and other officers who have or soon will have programs to support exceptional young scientists.
In addition, reviewers for the Beckman Young Investigator program applications reported that anonymizing author and institutional identity facilitated streamlined reviews and corresponding discussions, reduced overall workload, and decreased potential for burnout during that time.
“This study originated from an internal review and assessment of our program effectiveness, especially with regards to ensuring that we are advancing the most creative and novel scientific ideas for funding consideration, without introducing any bias or artificial constraints through our own internal practices. As stewards of funding for the most creative research ideas, it is essential that we are ourselves willing to take risks and challenge ourselves to innovate and address issues related to equity in our processes. We are proud of all of the awardees that have been selected in the Beckman Young Investigator program and the impact that they are having through their research. In sharing these results and details of our review process, we hope this information will be informative for others moving forward with evaluation of their application review processes with the goal of instituting more equitable practices, especially for those organizations with missions and funding programs similar to our own.
eLife is a peer-reviewed research journal that publishes high-quality content related to the life sciences. “Meta-Research: Blinding Reduces Institutional Prestige Bias During Initial Review of Applications for a Young Investigator Award” was authored by Anne E. Hultgren and Nicole M.F. Patras of the Arnold and Mabel Beckman Foundation in collaboration with Jenna Hicks of the Health Research Alliance; available under the persistent Crossref digital object identifier (DOI) 10.7554/eLife.92339.
About the Arnold and Mabel Beckman Foundation
Located in Irvine, California, the Arnold and Mabel Beckman Foundation supports researchers and nonprofit research institutions in making the next generation of breakthroughs in chemistry and the life sciences. Founded in 1977 by 20th century scientific instrumentation pioneer Dr. Arnold O. Beckman, the Foundation supports United States institutions and young scientists whose creative, high-risk, and interdisciplinary research will lead to innovations and new tools and methods for scientific discovery. For more information, visit beckman-foundation.org.
About Anne Hultgren, PhD
Dr. Hultgren joined the Arnold and Mabel Beckman Foundation as Executive Director and CEO in 2015. Previously, she was at the Department of Homeland Security, Science and Technology Directorate, working in chemical and biological defense technologies. She received her PhD in Physics and Astronomy from the Johns Hopkins University, and BA in Physics and Mathematics from Franklin and Marshall College.
About Nicole Patras
Mrs. Patras joined the Arnold and Mabel Beckman Foundation in 2014 and serves as the Senior Program Officer for the Beckman Young Investigator Program. She also leads the Foundation’s JEDI (Justice, Equity, Diversity, and Inclusion) Committee and oversees event planning for the Foundation’s annual Beckman Symposium. She holds a BA from the University of California, Santa Cruz and is a Certified Nonprofit Professional through Nonprofit Leadership Alliance.
About Jenna Hicks, PhD
Dr. Hicks joined the Health Research Alliance in 2023 as Project Lead of the Inclusive Grantmaking Initiative and was promoted to Assistant Director later that year. She came to HRA with a background in research (both biomedical and education research), program development, and evaluation. Prior to joining HRA, Dr. Hicks worked in graduate education administration at the University of Minnesota, where she developed, implemented, and evaluated professional development programming for biomedical graduate students and postdocs. Dr. Hicks received her PhD in biomedical sciences from the University of California, San Diego, and completed postdoctoral training in biology education research at the University of Minnesota.
Press Contact:
Kaerie Ray
Download a PDF of this Press Release:
See this press release on PRweb here.
