Peer review advisory committee 2 1 2010 pete morton and paul sheehy
1 / 31

Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy - PowerPoint PPT Presentation

  • Updated On :

Electronic Research Administration and the Receipt and Referral of Grant Applications. Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy. Context and Drivers. Continuous evolution of Processes and Policies Interdependencies of Business Processes.

Related searches for Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy' - HarrisCezar

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Peer review advisory committee 2 1 2010 pete morton and paul sheehy l.jpg

Electronic Research Administration and the Receipt and Referral of Grant Applications

Peer Review Advisory Committee


Pete Morton and Paul Sheehy

Context and drivers l.jpg
Context and Drivers Referral of Grant Applications

  • Continuous evolution of Processes and Policies

  • Interdependencies of Business Processes

Evolving needs of NIH’s extramural


  • IMPAC II is nearing 20 years old

  • Original System Structure Needs Refreshment

    • Lower Operations and Maintenance Costs

    • Greater Flexibility

New technologies are now available

  • IT Systems Development and Business Processes must be closely linked

  • Do not want to simply re-build the same thing without seeing if business processes would benefit from a change

Fundamental shift in the way IT is developed:

Three distinct activites l.jpg
Three distinct activites Referral of Grant Applications

IT Refreshment Strategy

Three distinct activities:

  • Develop New Hardware Infrastructure

    • Servers and SANs

    • Status: Complete

  • Develop New Software Infrastructure

    • Underlying IT applications and data tables

    • Status: Ongoing

  • Reengineer Applications “Evergreening”

    • Incorporate new infrastructure capacities

    • Refresh IT to incorporate new policies and business processes

    • Status: Starting

Evergreening a business function l.jpg
Evergreening a Business Function Referral of Grant Applications

  • General Strategy

    • Identify component business processes

    • Analyze current processes

      • Opportunity to identify and assess potential new support functions

    • Re-engineer IT systems

  • Approach

    • Create Business Process Models

      • “Current / As Is” state vs. “Future / To Be” state

    • Validate using focus groups of business subject matter experts

    • Model drives/directs IT re-engineering

  • Business Process Re-engineering (BPR)

    • Synthesis of Business Process Modeling and IT Reengineering

Business process reengineering l.jpg
Business Process Reengineering Referral of Grant Applications


  • Modeling ensures an excellent understanding of actions and rules between business processes and supporting IT systems

    • Establishes credibility

  • Modeling provides a “refreshed examination of business processes” before reengineering the IT support for those processes

    • Better visualization of inefficiencies and pain points

    • Holistic view reduces surprises

    • If the existing model is accurate and adequate, can proceed directly to IT re-engineering

  • Synthesis allows for faster, cheaper software development

    • Better defined target

Introduction bpr of csr receipt referral l.jpg
Introduction: BPR of CSR Receipt & Referral Referral of Grant Applications

  • As part of a larger effort to examine eRA reengineering, a pilot project to BPR CSR Receipt & Referral was initiated in June 2008

  • Mandate

    • Examine and document current referral business process re: IC and IRG assignment

    • Measure process performance: efficiency and quality of assignment

    • Understand referral process’ benefit to NIH’s scientific mission

    • Understand causes of sub-optimal process performance

    • Develop heuristic future state business process designed to improve performance and quality of assignment

Evergreening era l.jpg

Business Process Reengineering the Receipt and Referral Module of eRA: a Pilot

Evergreening eRA

Introduction division of receipt and referral drr l.jpg
Introduction: Division of Receipt and Referral (DRR) Module of eRA: a Pilot

  • DRR fulfills four business purposes:

    • Application assignment based on the scientific content of the application

      • Technical merit review group

      • Institute or Center

    • Application quality control

    • Policy enforcement

    • Facilitate and manage changes to assignment

  • All are critical to NIH’s scientific mission

Introduction business challenges l.jpg
Introduction: Business Challenges Module of eRA: a Pilot

  • Increase in volume of applications

  • Requirements for consistency, flexibility, transparency, equity, accountability

  • Aggressive implementation of electronic submission

  • Sometimes problematic IMPAC II migration to web

  • Increasing complexity of assignment decisions

    • Clinical, translational and multi-disciplinary research

    • Complexity is both review (subject matter, techniques, etc) and referral (relevance to multiple IRGs and/or IC missions)

      • e.g. A developmental disorder with cardiac & behavioral manifestations

Approach l.jpg
Approach Module of eRA: a Pilot

  • Validate current state business process models developed by DRR and Office of the Chief IT Architect

  • Establish metrics for the process

  • Identify areas of challenge for focus

  • Perform root cause analysis on these areas

  • Review potential solutions to issues including technology and process changes

  • Develop a new business process model

Methodology components l.jpg
Methodology: Components Module of eRA: a Pilot

Two methodologies needed:

  • Modeling: To document current processes and heuristic future process

    • Business Genetics XBML

  • Metrics: To measure efficiency and effectiveness of current process and provide input to future state processes

    • Lean Six Sigma

Methodology activities l.jpg
Methodology: Activities Module of eRA: a Pilot

  • Phase 1: Update of existing model

    • 2006 OCITA current state business process model used as a foundation

    • Project team, DRR staff, IRG chiefs reviewed, corrected/updated as necessary

    • Focus on IRG assignment, not IC

  • Phase 2: Obtaining process metrics

    • DRR staff and IRG chiefs provided estimates of the normal time taken for each major business process step

    • “Normal” defined as time required for 80% of applications

Evergreening era13 l.jpg

Findings Module of eRA: a Pilot

Evergreening eRA

Results process efficiency l.jpg
Results: Process Efficiency Module of eRA: a Pilot

  • 80% of Applications processed within one day

  • For this 80% of applications, metrics indicate overall efficiency is world class

    • Work/wait time distribution equals benchmarks regardless of industry

    • Wait time generated in part by policy

  • Conclusions

    • Efforts focused on reducing “wait-time” of grant applications will result in minimal improvement to referral process performance

    • Process changes should focus on the 20% of applications that take longer to process

    • However, there is still significant benefit to automating the less complex assignments

Results process efficiency 2 l.jpg
Results: Process Efficiency (2) Module of eRA: a Pilot

  • Remaining applications much more variable in time to refer

    • 20% of total applications

  • 2/3 due to problems with application itself

    • 13% of total applications

  • 1/3 are more complex to assign

    • 7 % of total

      • Scientific overlap between IRGs

      • Complexity of application

Results process efficiency of complex assignments l.jpg
Results: Process Efficiency of Complex Assignments Module of eRA: a Pilot

  • Factors contributing to complex assignment:

    • Overlaps/gaps in IC mission

    • Variation in sponsorship and use of activities

    • Complexities of locus of review

    • All require scientific judgment

  • Quality of assignment

    • Essential to validity of NIH Peer Review Process

    • BP and IT changes must retain and enhance quality of assignment going forward (i.e. not bouncing back from SRG)

  • Assignment delays impact review groups

    • May delay meetings

    • May require a new special emphasis panel

System changes issues with the era system l.jpg
System Changes: Issues with the eRA System Module of eRA: a Pilot

  • CSR staff discussions yielded the following issues with eRA:

    • Performance and reliability of Referral and Review modules

    • Fragility – Changes tend to introduce new problems

    • Security model does not allow others to view all referral data

    • Notes Field

      • Only place to track referral process e.g. Referral officer and IRG Chief and SRO comments

      • Lack of carryover between Receipt and Referral (RR) and Review (REV) modules

        • DRR uses RR, IRG Chiefs and SROs use REV, notes are not visible to each other

      • Lack of date/time/author stamping

      • Lack of carryover from ExitRamp

    • Release of applications and visibility to reviewers in IAR

    • Web-related issues

      • Inability to open more than one application window at a time

System changes drr priority list l.jpg
System Changes: DRR Priority List Module of eRA: a Pilot

DRR staff-identified priorities:

  • Support for fully electronic Awaiting Receipt of Application (ARA) notices

  • IRG Chief push to final assignment

  • Changes, fixes and improvements in the ACR system

  • Implement single 2 day extension of the 3 day assignment window

  • Support for automated comparison of application content

    • Identification of duplicate applications and revisions

    • COTS and custom KM products to compare current and previous submissions

Evergreening era20 l.jpg

Recommendations Module of eRA: a Pilot

Evergreening eRA

Future model changes l.jpg
Future Model: Changes Module of eRA: a Pilot

  • Suggested changes to existing processes:

    • Application Validation

      • Automate and move validations currently performed by CSR after application receipt to eSubmission process

      • Expand types of checks and validations applied to application prior to submission beyond the set limited by 424RR form set

      • Use these checks to either generate warnings for applicant or flag applications for review

      • Provide a mechanism for applications to validate their application prior to submission – “Pre-submission portal”

    • Incorporate the ability to create a special, customized workflow for specific applications that require handling outside the normal flow of operations

    • Modify the 3-day window for application release

System changes automation opportunities l.jpg
System Changes: Automation Opportunities Module of eRA: a Pilot

Other changes required in order to implement the “To Be” process

  • Provide system to support validation of an application against all and NIH business rules prior to submission to NIH “Pre-submission portal”

  • Support for a fully structured electronic “cover letter”

    • Could be implemented in Commons without need for new OMB form

  • Enhance ARWS to production ready status

    • Fully integrate with eRA, especially notes

    • Make ARWS decisions trainable

  • Implementation of a flexible, user-controlled workflow system

  • Expand the eSubmission business rules

  • Represent Funding Opportunity Announcement as electronic business rules

  • Represent of referral guidelines as electronic business rules

Suggested priorities l.jpg
Suggested Priorities Module of eRA: a Pilot

  • Immediate

    • Fixes for eRA issues, especially notes

    • Support IRG Chief push to final assignment and 3 day window extension (currently planned)

    • Implement Electronic Cover Letter in Commons

      • Consider providing an optional electronic form to be submitted as an interim measure

    • Fixes for ACR system

Suggested priorities cont d l.jpg
Suggested Priorities – cont’d Module of eRA: a Pilot

  • Medium Term

    • Implement Pre-Submission Portal

      • Expand and harden eSubmission business rules

    • Implement ARWS at production quality

    • Implement Electronic ARAs

    • Pilot application comparison tools

  • Long Term

    • Implement Flexible Workflow

    • Implement electronic business rules for:

      • Referral Guidelines (IC assignment)

      • Funding Opportunity Announcement (application validation)

Evergreening era25 l.jpg

Software Reengineering Phase Module of eRA: a Pilot

Evergreening eRA

Next steps era l.jpg
Next Steps (eRA) Module of eRA: a Pilot

Goal: Brief review/update of desired business changes resulting from BPR then rebuild IT support for R&R incorporating desired changes

  • Form core focus group

    • Includes staff from Division of Receipt and Referral as well as CSR IRG chiefs and IC representatives

    • First meeting scheduled for mid-February

Next steps era continued l.jpg
Next Steps (eRA) Module of eRA: a Pilot-- Continued

  • Working with core focus group, develop detailed requirements for new R&R module

    • Design to use new eRA software infrastructure

  • Initial detailed requirements documented –mid-summer

  • Develop prototype of new module – early fall

    • Update requirements as appropriate

Next steps era continued28 l.jpg
Next Steps (eRA) Module of eRA: a Pilot-- Continued

  • Begin iteration 1 development in fall, 2010

  • Work with focus group to test initial module and establish priorities for iteration 2 development – spring, 2011

  • Begin development of iteration 2 – summer, 2011

  • Work with focus group to test iteration 2 module and plan subsequent development if needed – late summer, 2011

  • Finalize and deploy new module into production – September 30, 2011

Evergreening era29 l.jpg

Where do we go from here? Module of eRA: a Pilot

Evergreening eRA

The next step review module l.jpg
The next step: Review Module Module of eRA: a Pilot

  • As BPM is complete for the Referral module and software reengineering is starting, the next opportunity is the review module.

  • Much bigger challenge

    • peer review policy and procedures are interpreted and executed in a variety of ways

    • Operational Divisions (OpDivs) under DHHS authority utilize NIH resources but don’t necessarily follow NIH policy

    • Agency Partners utilize NIH resources but don’t necessarily follow NIH policy

  • Initial recruiting is underway

    • Governance

    • Subject matter experts

Review by nih governance l.jpg
Review by NIH Governance Module of eRA: a Pilot

  • Trans-NIH Senior Staff

    • Extramural Activities Working Group (EAWG)

      • Working group of NIH Steering Committee with primary responsibility for extramural research and research training

  • Information Technology Working Group (ITWG)

    • Working group of NIH Steering Committee with primary responsibility for NIH's enterprise-level IT programs and investments.

  • Functional Areas

    • Extramural Program Management Committee (EPMC)

      • Senior leadership of IC extramural research and research training

  • Review Policy Committee (RPC)

    • Chiefs of Review Offices

  • Review Users Group (RUG)

    • SROs with interest in electronic research administration (NIH and IC)