There are lots of changes in the works related to the peer review process which we are communicating through updates to the NIH Web site on Enhancing Peer Review, policy notices in the NIH Guide for Grants and Contracts, presentations at professional meetings, etc.
The chart below compares current practices with new procedures on scoring of applications. (Original chart provided courtesy of NIAID Funding News (December 17, 2008), has been modified for use in the Nexus.)
|
Comparison of Existing and New Peer Review Processes |
||
|
Function |
Old |
New |
| Assignment of priority scores | Priority scores reflect reviewer judgment of a whole application: peer review criteria are unweighted and unrelated to the priority score. | Changed to preliminary impact score. Assigned reviewers also score each criterion (remains unrelated to the overall score) Before the review meeting, each reviewer and discussant assigned to an application will give a preliminary impact score for that application. The preliminary impact scores will be used to determine which applications will be discussed. For each application that is discussed, a final impact score will be given by each eligible committee member (without conflicts of interest). Each member’s impact score will reflect his/her evaluation of the overall impact that the project is likely to have on the research field(s) involved. |
| Each reviewer scores to one decimal place: 1.0 is best, 5.0 worst. | Each reviewer scores in whole numbers: 1 is best, 9 worst. | |
| Determination of priority scores | To create a raw score, reviewer scores are averaged and rounded mathematically to two decimal places, e.g., 1.34.The result is multiplied by 100 to give an overall priority score, e.g., 134.
The possible scores range from 100 to 500. |
The overall impact score for each discussed application will be determined by calculating the mean score from all the eligible members’ impact scores, and multiplying the average by 10; the overall impact score will be reported on the summary statement. Thus, the 81 possible overall impact scores will range from 10 – 90. (Overall impact scores will not be reported for applications that are not discussed.)Scoring with fewer rating options increases potential reliability and provides sufficient range and appropriate anchors to encourage reviewers to use the full scale. |
| Streamlined applications | Applicants get critiques from assigned reviewers. | Streamlined applications will receive scores on each criterion in addition to the reviewers’ critiques to help applicants assess whether or not they should resubmit an amended application |
| Determination of percentiles | Percentile range from 0.1 (best) to 99.5 (worst). Read How Percentiles Are Determined. | Percentiles range from 1 to 99 in whole numbers. Rounding is always up, e.g., 12.1 percentile becomes 13. |
| With almost 1,000 possible percentile rankings, few applications are ranked the same. | The new scoring system may produce more applications with identical scores (“tie” scores). Thus, other important factors, such as mission relevance and portfolio balance, will be considered in making funding decisions when grant applications are considered essentially equivalent on overall impact, based on reviewer ratings. | |
| Percentile base | NIH calculates percentiles using applications submitted for three review cycles. | Unchanged, except for the first year of the transition to the new review processes:
|
| Scores for Individual Criteria | Scores are not provided for individual critieria. | Before the review meeting, each reviewer and discussant assigned to an application will give a separate score for each of five core review criteria (Significance, Investigator(s), Innovation, Approach, and Environment). To improve information and transparency, for all applications, even those not discussed by the full committee, the scores of assigned reviewers and discussant(s) for these criteria will be reported individually on the summary statement. |
| Review Criteria | Five one-word criteria plus descriptive information. | One-word criteria unchanged; descriptions modified.Reviewers will consider each of the five review criteria in the determination of scientific and technical merit, and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact (i.e. a project that by its nature is not innovative may be essential to advance a field). |
Policy changes may be found in the following notices, or on the Enhancing Peer Review Web site.
- Review and Scoring Changes (NOT-OD-09-024 and NOT-OD-09-025)
- Changes to the Resubmission Policy (NOT-OD-09-003 and NOT-OD-09-016 )
- Recognition of New and Early Stage Investigators (NOT-OD-09-013, NOT-OD-08-121 and NOT-09-034)
0 Comments