Slide #1.

CSR’s Mission and Function and What’s New in Peer Review Martha M. Faraday, Ph.D. Scientific Review Officer Division of AIDS, Behavioral & Population Sciences Risk Prevention & Health Behavior IRG Psychosocial Risk & Disease Prevention Study Section Date : April 22, 2009 National Institutes of Health U.S. Department of Health and Human Services 1
More slides like this


Slide #2.

National Institutes of Health Office Officeof ofthe theDirector Director National NationalInstitute Institute on onAging Aging National NationalInstitute Institute on onAlcohol AlcoholAbuse Abuse and andAlcoholism Alcoholism National NationalInstitute Institute of ofAllergy Allergyand and Infectious InfectiousDiseases Diseases National NationalInstitute Institute of ofArthritis Arthritisand and Musculoskeletal Musculoskeletal and andSkin SkinDiseases Diseases National NationalCancer Cancer Institute Institute National NationalInstitute Institute of ofChild ChildHealth Health and andHuman Human Development Development National NationalInstitute Instituteon on Deafness Deafnessand andOther Other Communication Communication Disorders Disorders National NationalInstitute Institute of ofDental Dentaland and Craniofacial Craniofacial Research Research National NationalInstitute Institute of ofDiabetes Diabetesand and Digestive Digestiveand and Kidney KidneyDiseases Diseases National NationalInstitute Institute on onDrug DrugAbuse Abuse National NationalInstitute Institute of ofEnvironmental Environmental Health HealthSciences Sciences National NationalEye Eye Institute Institute National NationalInstitute Institute of ofGeneral General Medical MedicalSciences Sciences National NationalHeart, Heart, Lung, Lung,and andBlood Blood Institute Institute National NationalHuman Human Genome GenomeResearch Research Institute Institute National NationalInstitute Institute of ofMental MentalHealth Health National NationalInstitute Institute of ofNeurological Neurological Disorders Disordersand and Stroke Stroke National NationalInstitute Institute of ofNursing NursingResearch Research National NationalInstitute Instituteof of Biomedical BiomedicalImaging Imaging and andBioengineering Bioengineering National NationalCenter Center for forComplementary Complementary and andAlternative Alternative Medicine Medicine Fogarty Fogarty International International Center Center National NationalCenter Center for forResearch Research Resources Resources National NationalLibrary Library of ofMedicine Medicine National NationalCenter Centeron on Minority Health Minority Healthand and Health HealthDisparities Disparities Clinical ClinicalCenter Center Center Centerfor for Information Information Technology Technology Center for Scientific Review
More slides like this


Slide #3.

CSR Mission Statement To see that NIH grant applications receive fair, independent, expert, and timely reviews free from inappropriate influences - so the Institutes and Centers within the NIH can fund the most promising research.
More slides like this


Slide #4.

CSR Peer Review: 2008 Statistics • 77,000 applications received • 56,000 applications reviewed • 16,000 reviewers • 240 Scientific Review Officers • 1,600 review meetings
More slides like this


Slide #5.

Scientific Review Process Dual Review System for Grant Applications First Level of Review : CSR/Institute Review Scientific Review Group (SRG) (Study Section) Second Level of Review NIH Institute/Center Council
More slides like this


Slide #6.

CSR Review Divisions AIDS, Behavioral Basic and Integrative Physiological and Translational and euroscience, Development Pathological SciencesClinical Sciences Biological Sciences and Aging and Population Sciences Endocrinology, Biological Chemistry & Biobehavioral & Brain Disorders & Metabolism, Cardiovascular and Macromolecular Nutrition & Clinical Neuroscience Behavioral Processes Respiratory Sciences Biophysics Reproductive Sciences Surgical Sciences, Risk, Prevention& Molecular, Cellular & Biomedical Health Behaviors Bioengineering Sciences Developmental Neuroscience Immunology Imaging and & Technologies Population Sciences & Bioengineering Integrative, Functional & Epidemiology Infectious Diseases Musculoskeletal, Oral Cell Biology Cognitive Neuroscience & Microbiology And Skin Sciences Healthcare Delivery Genes, Genomes Emerging Technologies & & Methodologies Oncology 2 – Digestive, Kidney & & Genetics Training in Neuroscience Urological Systems Translational Clinical AIDS & Oncology 1 – Basic Related Research Vascular and Biology of Development Translational Hematology & Aging Interdisciplinary Molecular & Training
More slides like this


Slide #7.

Assignment to CSR Review Groups Within an IRG, applications are assigned for review to • Standing Study Sections when the subject matter of the application matches the referral guidelines for the study section • Special Emphasis Panels (SEPs) when the subject matter does not fit into any study section, or when assignment of an application to the most appropriate study section would create a conflict of interest. Also used for special mechanisms (e.g., fellowships, SBIRs, AREAs)
More slides like this


Slide #8.

When Preparing Your Application • Read the instructions • Never assume that reviewers “will know what you mean” • Refer to the literature thoroughly • State rationale of proposed investigation • Include well-designed tables and figures • Present an organized, lucid write-up • Remember to address human subjects, vertebrate animals, potential biohazards; these could affect your score • Obtain pre-review from faculty at your institution NIH Grant Writing Tips: http://grants.nih.gov/grants/grant_tips.htm
More slides like this


Slide #9.

Directing Your Application to a Specific Study Section • Peruse CSR Study Section Guidelines* to Identify a Possible Home for Your Application http://csr.nih.gov/ * Recently revised; alternative study sections listed in approximate order of degree of overlap • Submit a Cover Letter
More slides like this


Slide #10.

CSR Web Site: http://www.csr.nih.gov • About CSR • News and Reports • Peer Review Meetings • Resources for Applicants
More slides like this


Slide #11.

Role of Scientific Review Officer (SRO) Designated Federal official with overall responsibility for the review process • Performs administrative and technical review of applications to ensure completeness and accuracy • Selects reviewers based on broad input • Manages study section meetings • Prepares summary statements • Provides any requested information about study section recommendations to Institutes/Centers and National Advisory Councils/Boards
More slides like this


Slide #12.

WHOM DO I CONTACT?  Before review, contact the Scientific Review Officer in CSR  After review, contact your Program Officer in the NIH funding institute or center
More slides like this


Slide #13.

Pre-Meeting Review Process • Appropriate reviewers recruited by SRO; minimum of 3 “interactive” reviewers per application • Conflicts of interest identified • Applications made available to reviewers ~6 weeks prior to meeting • Critiques and preliminary scores posted by assigned reviewers on NIH web site at least 2-3 days prior to meeting • Critiques and preliminary scores (excluding conflicts) available to review group prior to meeting
More slides like this


Slide #14.

Where Do We Find Reviewers? • National Registry for Society-Recommended Reviewers • Successful applicants • Word of mouth Recommendations from study section members Recommendation from NIH IC staff • CRISP (crisp.cit.nih.gov) • PubMed • Scientific Conferences
More slides like this


Slide #15.

Traditional* Review Meeting Process • Upper half applications discussed: Reviewers are guided by specific review criteria Protections for Humans, Vertebrate Animals, Environment (Biohazard) may affect final score Assigned reviewers recommend scores for each application in upper half; all members not in conflict vote their conscience (outlier score policy pertains) Other considerations not affecting final score are discussed (e.g., budget, foreign applicants, resource sharing plans) • Lower half applications not discussed, not assigned an overall score * Aspects of this process will change in May, 2009 http://enhancing-peer-review.nih.gov
More slides like this


Slide #16.

Post Meeting Review Process • Scores are provided to investigators within 3 working days • Summary Statements for discussed and scored applications include Resume & Summary of Discussion, (largely unedited) critiques, and other recommendations (e.g., Budget) • Summary Statements for lower half (Not Discussed) applications receive (largely unedited) critiques and review criteria scores but no overall impact scores • All Summary Statements are made available within 30 days of meeting (10 days for new investigators’ R01s)
More slides like this


Slide #17.

What’s New in Peer Review?
More slides like this


Slide #18.

2008: The Year of Peer Review Enhancing Peer Review “Fund the best science, by the best scientists, with the least administrative burden…” Elias Zerhouni, MD, Former Director, NIH
More slides like this


Slide #19.

Recommendations
More slides like this


Slide #20.

Amended Applications: To speed the funding of meritorious science and minimize reviewer burden: •As of January 25, 2009, all original new applications (i.e., never submitted) and competing renewal applications will be permitted only a single amendment (A1). (A1)
More slides like this


Slide #21.

What’s New in Peer Review • New Investigators/Early Stage Investigators • Enhanced Review Criteria • Template-Based Critiques • Scoring Scale (9 point scale) Criterion Scoring Overall Impact Score 21
More slides like this


Slide #22.

New Investigators/Early Stage Investigators • New Investigator (NI): PD/PI who has not yet competed successfully for a substantial NIH research grant o For multiple PD/PIs-all PD/PIs must meet requirements for NI status • Early Stage Investigator (ESI): PD/PI who qualifies as a New Investigator AND is within 10 years of completing the terminal research degree or is within 10 years of completing medical residency (or equivalent) • • Applies only to R01 applications New Investigators/Early Stage Investigators will be clustered together for review 22
More slides like this


Slide #23.

Enhanced Review Criteria • Overall Impact: Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved • New Core Criteria Order: Significance Investigator(s) Innovation Approach Environment o Review criteria enhanced and expanded 23
More slides like this


Slide #24.

Template-Based Critiques Critiques Goal: To improve the quality of the critiques and to focus reviewer attention on the review criteria: •Provide clear, concise, and explicit information •Aid in identifying the strengths and weaknesses of each criterion
More slides like this


Slide #25.

Template-Based Critiques • Critique template contains a total of 18 boxes Reviewers should provide text for only those criteria that are applicable. 1. Significance 6. Resubmission 13. Overall Impact 2. Investigator(s) 7. Renewal 14. Budget and Period of Support 3. Innovation 8. Revision 15. Select Agents 4. Approach 9. Protection of Human Subjects 16. Applications from Foreign Organization 5. Environment 10. Inclusion of Women, Minorities, and Children 17. Resource Sharing Plan 11 Vertebrate Animals 18. Additional Comments to Applicant 12. Biohazards 25
More slides like this


Slide #26.

Template-Based Critiques • Goal: is to write evaluative statements and to discourage summarizing the application Comments should be in the form of bullet points or if necessary short narratives Do not record scores on the critique template The entire template is uploaded to IAR to become part of the summary statement. 1. Significance Please limit text to ¼ page Strengths    Weaknesses    26
More slides like this


Slide #27.

Scoring – 9 Point Scale Goal: To improve the transparency of the scoring process: • Score applications on five review criteria using a scale of 1-9. 1-9 • Preliminary overall impact score using 1-9 scale. scale Should not be the average of the criterion scores. Not Discussed applications will receive initial criterion scores from the three assigned reviewers
More slides like this


Slide #28.

Scoring Descriptions Impact High Impact Moderate Impact Low Impact Score Descriptor 1 Exceptional 2 Outstanding 3 Excellent 4 Very Good 5 Good 6 Satisfactory 7 Fair 8 Marginal 9 Poor Strengths/Weaknesses Weaknesses
More slides like this


Slide #29.

Clustering • NI/ESI R01 applications will be clustered together in review. ESI applications will not be separately clustered within the NI\ESI group. o NI/ESI applications will be identified for reviewers so there can be appropriate review in context of career stage. o Expectations of preliminary data and publication track record less than for established investigators.
More slides like this


Slide #30.

Order of Review Goal: Discuss applications in order of average preliminary score. Why: •Concern - variation of scores during different times of the meeting. One recommendation was to recalibrate scores at the end of the meeting . Solution: •Recalibrate “dynamically” throughout meeting.
More slides like this


Slide #31.

Order of Review • For calibration purposes: Begin meeting by discussing the best scored application (any activity code) o NI/ESI R01s clustered beginning of meeting o All other activity codes clustered if feasible (if at least 10 discussed (may include R03, R15, and R21s as a group that can be clustered)
More slides like this


Slide #32.

Order of Review Summary Discussion order is based on the average of the impact scores from assigned reviewers Final scores of discussed applications may differ from preliminary scores as recalibration happens dynamically
More slides like this


Slide #33.

Not Discussed Discuss ~ 50-60% of applications SRO will then ask if there are any other applications that panel wishes to discuss The remaining applications will not be discussed (applications receive criterion scores only) o Same after review of ~60% of SBIR applications
More slides like this


Slide #34.

Final Scores • Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90. • Percentiles will be reported in whole numbers.
More slides like this


Slide #35.

Summary Statements •Summary statement will be shorter and more focused. •Discussed applications will also have a summary of the panel’s discussion at the meeting. •ALL applications will be scored. scored Not discussed applications will receive criterion scores only.
More slides like this


Slide #36.

Recruiting the Best Reviewers  Move a meeting a year to the West Coast  Additional review platforms  Develop a national registry of volunteer reviewers  Searchable database with 4,000 reviewers  Provide tangible rewards for reviewers  No submission deadlines for chartered members of study sections (effective February 2008).  1574 chartered members used flexible deadlines during the last 6 months Provide flexible time for reviewers  Choice of 3 times/year for 4 years or 2 times/year for 6 years
More slides like this


Slide #37.

THANK YOU! This is CSR
More slides like this