240 likes | 337 Views
Strategy for quality measurement. Richard Nakamura. PhD CSR Advisory Council May 2014. In most production systems, there is a speed, cost, quality trade-off, but this is not a zero-sum game. We have just discussed speed and to some extent, cost. Effectiveness Review quality.
E N D
Strategy for quality measurement Richard Nakamura. PhD CSR Advisory Council May 2014
In most production systems, there is a speed, cost, quality trade-off, but this is not a zero-sum game. We have just discussed speed and to some extent, cost.
For Problem A, we have focused on:Attracting:…the best reviewers …the best Chairs …the best SROs
After we put together the best review committees, they must work in the best platform/format to determine merit of applications
Best reviewers -…recruit mostly senior scientists …funded awardees of NIH…w/ strong publications…positive review experience …respect from other reviewers…and diversity that aids the SRG
To get the best reviewers, we offer -…continuous submission…new forms of attendance…meetings on West coast
But -…we take away coffee…we make travel less convenient…we issue 1099 forms for all payments
Today, NIH wants to know how to measure quality so that we can systematically improve it.
To the extent that we have a standard, it has been to examine the results of grants ten years after award as evaluated by senior scientists. This has been done for the Pioneer awards (for 7 years) but provides a very slow feedback loop. Can we do better?
We would like to have measures available more immediately than 10 years after award that predict the longer outcome.
How do we get there?To begin with we need to have estimates of the reliability of review scoring (Rene Etcheberrigaray), so we can power our other measures of quality and interventions correctly.
We need evidence that our scoring approaches are not inappropriately biased – Monica Basco.
In measuring quality, CSR takes seriously the opinions of our scientific reviewers: Surveys – CSR quick feedback (Mary Ann Guadagno)Committee ranking within IRGS (Problem B and last Council)
Is the application ranking system working? (Amy Rubinstein)Do our committees favor conformity? (Seymour Garte)
What other approaches could help us evaluate quality? (actively pursued by other groups and not covered today)Bibliometrics (DPCPSI and CSR): Citations at individual, group levels Relative citation rating; h-index Concerns about citation manipulation
Review committee assignments and coverageNetwork analysis - Application and publication analysis of applicants and reviewersCreation of computer assisted assignment
AARR BBBP BCMB BDCN BST CB CVRS DKUS EMNR ETTN GGG HDM A System Level Representation of Scientific Interests IDM IFCN IMM MDCN MOSS OBT OTC PSE RPHB SBIB VH Other