Skip Navigation

Inter-rater reliability aims to bring standardization to ORAU scientific peer reviews

In some peer reviews, clear instructions on the basis for rating proposals may not be given to reviewers. As a result, the reviewers’ final ratings can leave much room for interpretation thanks to varying degrees of reviewer knowledge and experience‚Ķ all of which contribute to the final rating number.

For instance, the rating scale for a particular review is 1 to 5 (with 1 being the highest). One reviewer rates the proposal a 2, and a second reviewer rates the same proposal a 4 but notes “this is the best proposal” he’s ever seen. If the proposal was the best ever, why didn’t he rate the proposal a 1? And, was the range in their scores due to a difference in professional opinion or something else?

The answers to this question and others like it are what ORAU’s Scientific Peer Review Program is trying to uncover through a new initiative termed “inter-rater reliability” or IRR. Although IRR may have a different meaning in another industry, for ORAU the goal of IRR is to establish standards or a level of agreement among reviewers when rating a proposal.

The need for IRR proves even more important for ORAU’s customers who must award/fund proposals in “rank order” or based on the highest ranking. The goal is to increase the likelihood that the reviewer’s score for each proposal in a review has the same meaning.

To discuss and test the IRR concept, ORAU recently conducted a working meeting with experts in a range of disciplines including statistics, data analysis and human factors.

The meeting, funded by ORAU’s internal research and development, incorporated questions like these as part of the discussion: “Should the rating numbers include a middle, neutral number such as 3?” or “Should only even numbers be used which ‘forces’ reviewers to give a more focused rating?”

At the meeting’s conclusion, ORAU walked away with a report from the experts that included recommendations and suggested next steps.

Although there appears to be no definitive answer to implementing IRR, the concept was eagerly received by the experts. ORAU is currently studying the outcomes report to determine the next best course of action.

More peer review success stories | Scientific peer review home