WG5 P02 Proposal 2014 Qualification of Standard Scripts

1. WG5 P02 Proposal 2014Qualification of Standard Scripts

2. Proposal through CSS 2014http://www.phusewiki.org/wiki/index.php?title=File:FDA_Scrips.ppt • Anyone should be able to submit a script, according to a check list • Categorize scripts according to complexity • Complexity: low, medium, high, software • Output: tabulated data, analysis data, table, figure, listing • Metadata for script should indicate • Type of output: tabulated data, analysis data, table, figure, listing • Study design: parallel, crossover, etc • State of qualification: ?

3. Proposal through CSS 2104 • Test data • Overall project should have minimum test data (SDTM & ADaM) • Scripts can propose new test data, must pass (Data fit? Open CDISC?) • Share program to produce test data, never binary test data • 2 levels of qualification to match script complexity/output • Light vs. Heavy qualification • Common elements include • header • good programming practices • clearly declared scope of script (e.g., study design(s)) • test data matches scope & passes "FDA Data Fit" assessment (?) • documentation inputs/outputs/dependencies/usage

4. Proposal through CSS 2104 • Heavy qualification • Beta package includes minimal elements for contribution • Specification & Documentation (could be in pgm header) • Test data (Data Fit? or Open CDISC or other, as appropriate) • Tests & Expected results defined • Peer Review: GPP, Specs & Docn reviewed, Tests reproduced • Draft • Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) • Test • Peer Review: Write qualification report, incl. log/output from tests • Final

5. Proposal through CSS 2104 • Light qualification • Beta package includes skip if >1 yr production usewithout ERROR • Draftminimal elements for contribution • Specification & Documentation (could be in pgm header) • Test data (Data Fit? or Open CDISC or other, as appropriate) • Tests & Expected results defined • Peer Review: GPP, Specs & Docn reviewed, Tests reproduced • Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) • Test • Peer Review: Write qualification report, incl. log/output from tests • Final

6. Proposal through CSS 2104

7. Proposalmeaningful terms in blueQualificationhttp://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx • Qualification Instructions (see embedded template ð) • Certificationphase of Qualification applies to new scripts and tests • Confirmation phase applies to updates of existing scripts • States: Contributed, Development, Testing, Qualified • Roles • Contributor: Anyone with appropriate skills & interests • Developer: CSS Working Group 5 volunteer familiar with objectives** • Tester: CSS WG 05 volunteer familiar with objectives** • Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment • Reviewer: Author of white papers, designers of script targets ** suggests a quick-start onboarding page in CSS Phusewiki

8. ProposalQualification • Metadata for script should indicate • Whitepaper & output ID • Programming language & version (e.g., R 3.1.1) • Type of output: tabulated data, analysis data, table, figure, listing • Study design: parallel, crossover, etc • State of qualification: Contributed, Development, Testing, Qualified • Test Data requirements • available as a program or script (text, not binary) • based on expected standards (SDTM, ADaM) • data requirements clearly & accurately specified for each script • include expected results from these data for defined tests/checks

9. ProposalQualification • End-user Objectives • Clear overview of purpose and resources • Inspire confidence from first sight • Ease of use, clear messaging from first run • Reproducible results • Consistency of scripts, learning first one makes remaining familiar • Ease of converting users to contributors • Contributor Objectives • Standardize routine steps • Modularize routine components • Automate testing, issue identification • Centralize & consolidate information & results

10. ProposalQualification • TransitionsContributed is the original State of all scripts • to Development, checklist includes by Developer & Reviewer • R & D confer on suitability of contribution. Suitable starting point?[ may require analysis details, specs, from contributor ] • D reviews components • D works with Contributor to complete minimum components[ including Test Data and Coverage of defined tests ] • D adds standard parameter, dependency checking • D writes Qualification instructions .docx (see template, above) • to Testing, checklist includes by Tester • Review Qualification instructions, consider coverage of tests • Execute Qualification instructions • Work with Developer to complete execution successfully

11. ProposalQualification • Transitionscontinued • to Qualified by Tester & Environment Tester & Reviewer • T updates posted test outputs from certification/confirmation • E updates local tests and executes (posting PASS/FAIL results) • R confirms script output matches intention & qualification process covers important elements and considerations. Also provides user (rather than technical) feedback? • Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

12. ProposalQualification • Efforts Required • Finalize Qualification states, roles, workflow, checklists, and templates • Design test structure in google code • Develop scripts that will allow Environment Testing • Develop general components (e.g. parameter, dependency checking) • Identify Environment Testers based on • Host environment • SAS or R version • Identify opportunities to automate qualification. E.g., • Docx format for Qualification instructions is not easily machine readable • Environment Testers to post results back as machine readable • Script green-light/red-light qualification matrix on Phusewiki

13. Doing now what patients need next