The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome

Abstract

When distributing grants, research councils use peer expertise as a guarantee for supporting the best projects. However, there are no clear norms for assessments, and there may be a large variation in what criteria reviewers emphasize – and how they are emphasized. The determinants of peer review may therefore be accidental, in the sense that who reviews what research and how reviews are organized may determine outcomes. This chapter deals with how the review process affects the outcome of grant review. It is a reprint of a study of the multitude of review procedures practiced in The Research Council of Norway (RCN) in the 1990s. While it is outdated as an empirical study of the RCN, it provides some general insights into the dynamics of grant review panels and the effects of different ways of organising the decision-making in the panels. Notably, it is still one of the few in-depth studies of grant review processes based on direct observation of panel meetings and full access to applications and review documents. A central finding is that rating scales and budget restrictions are more important than review guidelines for the kind of criteria applied by the reviewers. The decision-making methods applied by the review panels when ranking proposals are found to have substantial effects on the outcome. Some ranking methods tend to support uncontroversial and safe projects, whereas other methods give better chances for scholarly pluralism and controversial research.