2. Why Develop A New Instrument?. Development of clinical treatments requires a suitable instrument to measure aspects particular to the disease and population. Previously published instruments may:Not adequately measure intended treatment effect e.g. new research areaNot adequately reflect t
1. Development of Patient-Centered Questionnaires FDA/Industry Workshop – Washington DC 2005 Cindy Rodenberg, Ph.D
Procter & Gamble Pharmaceuticals
2. 2 Why Develop A New Instrument?
Development of clinical treatments requires a suitable instrument to measure aspects particular to the disease and population.
Previously published instruments may:
Not adequately measure intended treatment effect – e.g. new research area
Not adequately reflect the patient population
Use language that is dated
Not translated or linguistically harmonized
3. 3 Population-specific: Is that really important?
4. 4 Three Steps of Instrument Development
Preliminary Instrument Development
Face and content validity
Qualitative Content Validation
Translation and linguistic validity
Reliable, valid, and sensitive
to treatment effect
5. 5 Step 1: Preliminary Instrument Development Item Generation - Identify characteristics of the disease and ensure adequate sampling of content to provide valid measurement
Face validity – Appears relevant to the outcome intended to capture.
Content validity – Sampling of items, presentation, and measurement of all aspects of the disease or states relevant to the patient.
6. 6 Importance of content validity
Study specific instruments may be biased “…if since the sponsor decides what it wants to ask, and thus can emphasize areas where the product should excel and deemphasize potentially troublesome areas…”
Smith, N. Quality of Life Studies from the Perspective of an FDA Reviewing Statistician, Drug Information Journal, 1993; 27: 617-623
7. 7 Methods – Item Generation Experts
Menopausal Sexual Interest Questionnaire (MSIQ) – 10 item questionnaire judged to capture key components of sexual desire and response
Literature review/Existing questionnaires
Mini Mental from the WAIS Intelligence Test
Patient-centered approach - Focus groups and individual patient interviews
Profile of Female Sexual Function (PFSF)
8. 8 Patient Centered Approach
Focus groups and individual interviews
Structured, Semi-structured, or Unstructured
Subjects selected to ensure adequate representation across relevant population subgroups
Sample until ‘data saturation’ – in other words, no new information.
Approximately 5-10 subjects per subgroup
9. 9 Response Levels Direct Estimation Method – Directly quantify magnitude of a trait
Visual Analog Scale: Arthritic Pain?
10. 10 Step 2: Qualitative Content Validation Elimination and refinement to ensure items:
Represent patients’ symptoms/experiences
Have clear and unitary meaning
Have similar meaning across translations
Cognitive Interview Technique
Interviewer probes on thought process used by patient in determining response
11. 11 PFSF Content Validation
12. 12 Step 3: Quantitative Statistical Approaches
Methods for item reduction and domain identification
Methods for assessing reliability and validity
13. 13 Item Reduction and Domain Identification Data quality assessment
Missing data frequencies
Item Frequency Distributions – Floor/Ceiling effects
Eliminated item - “We had sex any time and any place”
Ability to detect known group differences
T-test - parametric
Area Under an ROC Curve - nonparametric
14. 14 Item Reduction and Domain Identification Principle components analysis and Factor analysis:
Unidimensional domains – measuring some facet of same underlying construct
Items loading across multiple factors or small loadings (<0.4) across all factors are eliminated
Multi-trait analyses – Item-total correlations to assess:
Convergent/Divergent Validity – Items correlate more with their own domain than with other domains
Software available – Multi-trait Analysis Program
15. 15 Item Reduction - Factor Analysis
16. 16 Evaluating an Instrument - Validity Validity - Property of measuring what is intended to be measured
Content validity – adequate sampling and representation of relevant disease characteristics
Concurrent validity – Good correlation with a “Gold Standard”
Construct validity – Extent to which an instrument measures the underlying constructs purported to represent
17. 17 Construct Validity Convergent validity - Instrument correlates with other measures off related aspects
Divergent validity - Instrument is not correlated with measures on unrelated aspects
Ability to distinguish groups known to differ
18. 18 Treatment Sensitivity Decreases in Distress significantly greater on Testosterone than on Placebo
19. 19 Desire score differentiates normal libido women from low libido women
20. 20 Evaluating an Instrument - Reliability Reliability – agreement between two or more measures of the same thing
Test-retest reliability – reproducibility of score over separate measurement occasions
Pearson correlation coefficient
Intraclass correlation coefficient
Internal consistency reliability – homogeneity of items within a domain
21. 21 References Fayers P.; Hays R. Assessing QoL in Clinical Trials. Second edition: 2005.
Streiner, D. L.; Norman, G. R. Health measurement scales: A practical guide to their development and use 2nd Ed. Oxford University Press: 1995.
Shrout, P.E.; Fleiss, J. L. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 1979, 86, (2), 420 – 428.
Nunnally, J. C.; Bernstein, I. H. Psychometric Theory, 3rd ed. McGraw-Hill: New York, 1994.
Juniper, E. F.; Guyatt, G. H.; Jaeshke, R. How to develop and validate a new health-related quality of life instrument. In Quality of life and pharmacoenconmics in clinical trials, 2nd Ed., Soilker, B., Ed.; Lippincott-Rqven: Philadelphia, 1996; 49-56.
Rodenberg, C.A.; Kuznicki J.; Yiu G. Instrument Development and Validation. In Encyclopedia of Biopharmaceutical Statistics. 2002