Jump to comment:

• Is peer review in Injury Prevention better than random?
  Eric M. Ossiander
  Published on: 13 May 2016

• Published on: 13 May 2016

  Is peer review in Injury Prevention better than random?

  • Eric M. Ossiander, Epidemiologist

  Dear Editor,

  In an editorial [1], you presented data from a study of peer reviews of 20 randomly selected papers submitted to Injury Prevention. Each paper was independently reviewed by three reviewers, who score...

  Show More

  Dear Editor,

  In an editorial [1], you presented data from a study of peer reviews of 20 randomly selected papers submitted to Injury Prevention. Each paper was independently reviewed by three reviewers, who scored the paper as low, medium, or high on four dimensions: significance, appropriateness, science, and writing. The editorial concluded that agreement among the reviewers was good. In fact, according to the data presented, agreement on some dimensions was barely better than would be expected by chance, and on only one dimension was agreement significantly better than would be expected by chance in a sample of 20 papers.

  The expected agreement score among three reviewers, assuming random assignment of scores, can be calculated analytically. In your analysis, you defined the agreement score as 100% if all three reviewers agreed, 66% if two reviewers agreed, and 0% if all three gave different scores. The mathematical expected value of the agreement score is 100%×P(all 3 reviewers agree)+66%×P(2 agree)+0%×P(none agree), where P(-) denotes the probability of an event. Under random assignment, the probability that 3 reviewers agree is 1/9, that 2 agree is 6/9, and that none agree is 2/9 (see appendix). Therefore, the expected agreement score would be 100%×1/9+66%×6/9+0%×2/9 = 55.1%. The reported agreement scores for the significance (56%), science (60%), and writing (62%) dimensions are barely better than this.

  This problem also lends itself well to a computer simulation. I wrote a program to randomly generate scores on a three-point scale for each of 3 reviewers. The program calculated an agreement score as described above. This was repeated for 20 papers, and the mean agreement score was calculated. Then this was repeated 10,000 times to evaluate the distribution of mean agreement scores in samples of size 20. The computer simulation does not require any probabilistic calculations and is independent of the analytical approach described above. The simulation results were: mean score: 55.1%; SE: 7.0%; 2.5th and 97.5th percentiles (i.e. an interval which contains 95% of the means from the simulated distribution): (41.3%, 67.9%). Only the agreement score you reported for appropriateness (69%) was outside this interval.

  These results do not necessarily suggest that all three reviewers perform no better than chance. If each paper has a particular score which is correct, and one reviewer always scores each paper correctly, while the other two reviewers score papers randomly, the agreement among the three reviewers would be the same as if all three reviewers scored papers randomly. However, this makes a large difference in the agreement between the reviewers and the correct score. If all three reviewers score randomly, the probability that at least two will agree with the correct score is only 7/27, and the probability that at least two will agree on an incorrect score is 14/27. If one reviewer scores correctly, and the other two are random, the probability that at least two will agree with the correct score is 5/9, and the probability that two will agree on an incorrect score is 2/9.

  If two reviewers always score papers correctly, and the third scores randomly, the expected agreement score would be 77.3%.

  Your study suggests that agreement among Injury Prevention peer reviewers may be little better than random. As a reader, I find this discouraging. As a sometime author (not in Injury Prevention), I am sorry to say that I do not find it surprising.

  Appendix

  There are many ways to calculate the probability that 3, 2, or 0 reviewers agree with each other under random assignment. The method that makes this calculation easiest to understand is to enumerate all the possible ways that a paper can be scored (hopefully, this will resolve the disagreement between experts mentioned in the editorial). Since each of 3 reviewers can assign any one of 3 scores, there are 3×3×3 = 27 different ways to score each paper. Say we denote the scores as 1, 2, or 3, and that we denote the 3 reviewers’ scores for each paper as a triplet in which the scores for reviewers A, B, and C are in the first, second, and third positions respectively. The 3 combinations in which all 3 reviewers agree are 111, 222, and 333. The 18 combinations in which 2 agree are 112, 113, 121, 131, 122, 133, 221, 223, 212, 232, 211, 233, 331, 332, 313, 323, 311, and 322. The 6 combinations in which none agree are 123, 132, 213, 231, 312, and 321. Therefore the probability that all 3 reviewers agree is 3/27 = 1/9, that 2 agree is 18/27 = 6/9, and that none agree is 6/27 = 2/9. Note that the probabilities sum to 1, as they must.

  References

  [1]Pless IB. When reviewers disagree.InjPrev 2006;12:211.

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top