1 / 16

Standard Scripts Qualification Proposal - Updated 2014

This proposal aims to standardize the qualification process for scripts, including categorizing scripts by complexity, defining test data requirements, and providing clear documentation. It also outlines the roles and states involved in the qualification process.

clori
Download Presentation

Standard Scripts Qualification Proposal - Updated 2014

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Standard Scripts Project 2 2014 - Proposal for Qualification of Standard Scripts

  2. Main Sections • Summary of prior proposal, 2013 • Updated proposal, July 2014

  3. Main Sections • Summary of prior proposal • Concepts, definitions & metadata • Test data considerations • Heavy vs. Light qualification • Updated proposal

  4. Proposal from 2013 • Anyone can submit a script, according to a check list • Categorize scripts by complexity • Complexity: low, medium, high, software • Output: tabulated data, analysis data, table, figure, listing • Metadata for script should indicate • Type of output: tabulated data, analysis data, table, figure, listing • Study design: parallel, crossover, etc • State of qualification: ?

  5. Proposal from 2013 • Test data • Overall project should have minimum test data (SDTM & ADaM) • Scripts can propose new test data, must pass (Data fit? Open CDISC?) • Share program to produce test data, never binary test data • 2 levels of qualification • Light vs. Heavy: according to complexity of script & output • Common elements include • header • adhere to good programming practices • clear scope of script (e.g., study design(s)) • test data matchescope & pass "FDA Data Fit" assessment (Open CDISC?) • documentation available for: usage, inputs, outputs, dependencies

  6. Proposal from 2013 • Heavy qualification • Beta package includes: minimal elements for contribution • Specification & Documentation (could be in pgm header) • Test data (Data Fit? or Open CDISC?) • Tests & Expected results defined • Peer Review: GPP, Specs & Docn reviewed, Tests reproduced • Draft • Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) • Test • Peer Review: Write qualification report, incl. log/output from tests • Final

  7. Proposal from 2013 • Light qualification • Beta package includes skip if >1 yr production usewithout ERROR • Draftminimal elements for contribution • Specification & Documentation (could be in pgm header) • Test data (Data Fit? or Open CDISC or other, as appropriate) • Tests & Expected results defined • Peer Review: GPP, Specs & Docn reviewed, Tests reproduced • Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) • Test • Peer Review: Write qualification report, incl. log/output from tests • Final

  8. Proposal through CSS 2104

  9. Main Sections • Summary of prior proposal • Updated proposal • Objectives • Definitions: Qualification, States, Roles • Metadata and Test data • State Transitions

  10. Proposal 2014 - Objectives • For End-users • Clear overview of resources available, incl. purpose & state of each • Inspire confidence from first user experience • Ease-of-use: • clear messaging from scripts • Reproducible results in user's own environment • Consistency of scripts, learning first one makes remaining familiar • Ease of converting users to contributors • For Contributors & Standard Scripts Team • Standardize workflows and checklists • Modularize routine components (e.g., FUTS for dependency checking?) • Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility) • Centralize & consolidate information & results

  11. Qualification Proposalmeaningful terms in bluehttp://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx • Qualification Instructions (see embedded template ð) • Certification phase of Qualification applies to new scripts and tests • Confirmation phase applies to updates of existing scripts • States: Contributed, Development, Testing, Qualified • Roles • Contributor: Anyone with appropriate skills & interests • Developer: CSS Working Group 5 volunteer familiar with objectives** • Tester: CSS WG 05 volunteer familiar with objectives** • Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment • Reviewer: Author of white papers, designers of script targets** ** suggests a quick-start onboarding page in CSS Phusewiki

  12. Qualification Proposal • Metadata for script should indicate • Whitepaper ID & output ID • Programming language & version (e.g., R v3.1.1, SAS v9.4) • Type of output: tabulated data, analysis data, table, figure, listing • Study design: parallel, crossover, etc • State of qualification: Contributed, Development, Testing, Qualified • OS is not included, since should be indicated in OS compatibility report • Test Data requirements • available as a program or script (text, not binary) • based on expected standards (SDTM, ADaM) • data requirements clearly & accurately specified for each script • include expected results from these data for defined tests/checks

  13. Qualification Proposal • Transitions"Contributed" is the original State of all scripts • to Development, checklist includes by Developer & Reviewer • R & D confer on suitability of contribution. Suitable starting point?[ may require analysis details, specs, from contributor ] • D reviews components (checklist to be completed) • D works with Contributor to complete minimum components[ including Test Data and Coverage of defined tests ] • D adds standard parameter, dependency checking • D writes Qualification instructions .docx (see template, above) • to Testing, checklist includes by Tester • Review Qualification instructions, consider coverage of tests • Execute Qualification instructions • Work with Developer to complete execution successfully

  14. Qualification Proposal • Transitionscontinued • to Qualified by Tester & Environment Tester & Reviewer • T updates reference test outputs from certification/confirmation • E updates & executes local tests (posting PASS/FAIL results) • R confirms script output matches intention • R confirms qualification process covers important elements and considerations. • R also provides user (rather than technical) feedback? • Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

  15. Qualification Proposal • Efforts Required • Top priority • Finalize Qualification states, roles, workflow, checklists, and templates • Next priorities • Design test structure in google code • Develop scripts that will allow Environment Testing • Develop general components (e.g. parameter, dependency checking) • Identify Environment Testers based on • Host environment • SAS or R version • Identify opportunities to automate qualification. E.g., • Environment Testers to post results back as machine readable • Script green-light/red-light qualification matrix on Phusewiki

More Related