Carl Sagan, who advocated scientific skeptical inquiry and use of the scientific method, would likely be shocked by some of what passes for science these days. Everything from the health benefits or harms of coffee to whether childhood vaccinations cause autism are either fodder for online clickbait or vehemently called into question by people with little to no scientific background.
In 2020, Cagle and her team facilitated the reviews of more than 3,202 proposals using 4,175 reviewers.
Living in a culture of scientific skepticism means it’s critical to maintain integrity throughout the scientific process. “Integrity is number one,” said Keri Cagle, director of peer review for ORAU and ORISE. “That means the products that we are putting together and the services we provide need to be of the highest quality. If we’re not providing quality service, how can we justify the integrity of the review?”
Cagle adds that peer review and evaluation is the unsung hero of the research funding and proposal evaluation process, yet it is critically important to ensuring available research dollars are spent wisely.
“Peer review really does not get the credit it deserves because so much of the process happens behind the scenes,” she said. “Peer review is not an evaluation of the research being proposed. It’s an evaluation of research funding proposals, or research project proposals, from researchers who are requesting funding for their work. This may include allocation of scarce federal resources like personnel placements, equipment usage or purchase, and use of scientific facilities.”
All federal research projects are mandated by law to be peer reviewed, Cagle said. ORAU has conducted peer reviews for a number of governmental agencies and other institutions, including the Department of Energy, Florida Department of Health, Pennsylvania Department of Health, Nazarbayev University in Kazakhstan, and more.
ORAU’s peer review team manages the entire life cycle of the peer review process. This includes facilitating and coordinating advisory committee meetings and workshops where decisions are made about which topics should be researched and funded. Then the team collects proposals using PeerNet, ORAU’s proprietary peer review management portal or customer evaluation system. They then identify and recruit subject matter experts from ORAU’s database of more than 19,000 global experts. Finally, the team coordinates and executes the peer review meeting, whether in person or via webinar, and provides results who the customer, who will use the information to make funding decisions.
The entire process helps ensure our customers get the very best value for the research funding dollars they have available, Cagle said. Throughout the process, the peer review team remains focused on maintaining integrity. In 2020, Cagle and her team facilitated the reviews of more than 3,202 proposals using 4,175 reviewers.
Avoiding Conflict of Interest
Conflict of interest is another important issue. “We make sure we minimize any conflict of interest by vetting actual and potential conflicts between reviewers and proposals before they are officially assigned to the review,” said Jim Malone, research services section manager. Conflicts can be personal or organizational.
On the personal side, “we check to see if the reviewer has any co-publications with the principal investigator on the proposal, or have they worked together on a research grant in the last five or 10 years. Have the reviewer and the PI [principal investigator] had a mentor/mentee relationship, like in graduate school? Are they friends or relatives?” Malone said.
“We do what we can on the front end, but when the reviewer is in the peer review system, such as ORAU’s PeerNet, and sees the proposal, they are required to state if they do or do not have a conflict of interest with each proposal they are assigned. Some need to indicate that they have a conflict that maybe only they would know about, like the principal investigator is a brother-in-law or something,” he said.
According to the Department of Energy, conflicts of interest may arise under several circumstances, including current or known future work for applicants, personal benefits that a reviewer, employer or family member may receive, previous involvement with projects/programs that reviewers are asked to review, and related financial interests. In some circumstances, avoiding conflicts of interest can be a challenge in scientific fields where the number of researchers is small, like high energy physics or nuclear fusion research, said Jody Crisp, peer review group manager.
“Some fields are so specialized, most researchers are acquainted with each other in some capacity. Fortunately the vast majority of reviewers look beyond that familiarity and act accordingly in their evaluations. Reviewers are incredibly dedicated to ethically advancing their scientific areas through the peer review process,” stated Crisp. “The good news is our reviewers and most all scientists want the best for their field, they want the best for the science. It’s not about what they get out of it,” said Crisp.
Also critical to ensuring scientific integrity is the prevention of plagiarism. Plagiarism is the practice of using someone else’s work without giving attribution to the source. Ideas, words and work can be plagiarized, and some researchers may even plagiarize themselves if they do a lot of work in the same subject area.
To illustrate the importance of this issue, Crisp related an incident at a panel review when a proposal being evaluated contained work plagiarized from one of the reviewers. “The reviewer recognized his own work in the proposal, which eliminated the proposal from funding consideration and had significant consequences for the plagiarizing researcher,” said Crisp.
“Turns out one of the co-principal investigators who wasn’t at the review plagiarized some of the reviewer’s materials, and that pretty much brought the peer review to a screeching halt,” Crisp said.
While plagiarism in the peer review process may be difficult to detect, ORAU is committed to scientific integrity in its own research enterprise, which ultimately helps protect work we do for and with customers. To that end, ORAU uses iThenticate plagiarism detection software on any research articles written by ORAU employees. It is also used by most major universities and funding agencies, as well as by many journals and publishers.
Malone said iThenticate scans the document for comparison against billions of web pages and previously published research. The results produce a percentage of similarity to other work. Matching common words and phrases across hundreds of other documents is not a big deal.
“However, if there is a whole paragraph that is exactly the same as a previously published work, and it hasn’t been properly cited or in quotes, that’s a problem,” Malone said. In the rare event of such a situation, Malone said the author gets pre-publication diversion, or the opportunity to manage the issue before the document is sent for publication where it will be discovered. “You don’t want it to get out there and be published and someone else come through and say you plagiarized,” he said.
Ensuring Diversity of Thought
While we often talk about diversity in the context of workplaces, diversity of thought is very important in the peer review process.
“One of the things our research services team looks for when they are recruiting reviewers is different trains of thought,” said Meredith Goins, group leader for scientific assessment. “You can be in high energy physics and believe that one model works better than another and still be open minded, for example.”
“We do try to get a diversity of people from different organizations and universities,” Malone said. “You don’t want everyone in the review to come from the same place, because they often share similar perspectives. We want reviewers to honestly express their opinion from the perspective they have as individuals.”
Goins added that it’s important for the research process for proposals to be discussed robustly in peer review. “If there is no diversity of thought, discussion of a project’s merits cannot happen very well,” she said. “If a research proposal is deemed worthy of funding, it should be because it was deemed worthy through the process. Our customers are counting on us to ensure that happens.”
Avoiding Implicit Bias
Implicit bias stems from the unconscious assumptions individuals make when they are confronted with a resume, curriculum vitae or project proposal. While implicit bias is common in all fields, Goins said researchers and reviewers have to purposefully look past their own biases when reviewing research proposals.
“People make unprovoked judgements on others due to their name, location, or where an individual went to school,” she said. “Researchers have to be intentional about seeing beyond any potential biases they have on gender, ethnicity, disabilities, nationality and institutions and only consider the quality of the science being presented.”
The peer review team is researching implicit bias training programs and will make recommendations on best practices to further ensure implicit bias issues are avoided in the peer review process.
Avoiding conflict of interest, preventing plagiarism, ensuring diversity of thought and avoiding implicit bias are important issues that have to be addressed in every peer review. In the end, avoiding or preventing these issues not only makes for a better review, it makes for better science.