Practical Problems

Practical Problems Week 6Lecture H615 - Advanced Evaluation & Research Design By: Krissi Hewitt and Tassnym Sinky

The Ethics of Experimentation Assign groups of 4-5 for activity: • Ethical Codes • Informed Consent • IRBs • Legal Issues

Ethics of Experimentation in Practice In regard to the human subjects in this study... • Consider the following for your assigned topic as you watch the video clip… • What is currently in place? • Violations? • Ways the process could be improved?

Ethics of Experimentation in Practice • Discuss in your assigned groups • Come up with presentation points about your topic 5 minutes

Practical Problems: Recruitment • Participants affect construct and external validity • Can be difficult to locate some participants • Can get lower number than expected/desired for power • Volunteers vs. Non - different outcomes

Practical Problems: Recruitment If you can’t find your participants you can: • Conduct pre-surveys to locate • Conduct pipeline studies • Pilot tests of solicitations • Hire trained outreach specialists • Hire aggressive recruiters • Study potential barriers to enrollment

Practical Problems: Recruitment If you don’t have enough participants you can: • Extend time frame • Intensify outreach • Alter eligibility (interactions may occur) • Reduce prop. assigned to treatment • Terminate experiment

Methods of Random Assignment • Simple RA • Restricted RA to force equal sample sizes • Restricted RA to force unequal sample sizes • Batch • Trickle process • Adaptive • RA from matches or strata

Develop a Random Assignment Method • Case study on sheet of paper • As a group, come up with a random assignment method for your case. • Present your case and method to the class Questions to consider: What method(s) would you use? How does it work? What drawbacks/benefits are there to this method?

Practical Problems: When Pretest Means Differ Matching • before randomization • match on a variable • can match on pretest scores (preferred) Stratifying • more units than conditions • Ex. males and females are RA to treat/control

Matching and ANCOVA • covariate - variable not part of the experiment but that influences the outcome (DV) • Added into regression model first to see effect on IV after. • Reduces within-group error variance • Eliminates confounding variables • Limit # of covariates because high number inc. degrees of freedom.

Matching and Stratifying • Both increase statistical power • Recommended for small sample sizes Caveats: • Matching on unrelated variable can dec. statistical power without benefit • Should be done PRE RA

Successful Implementation of RA • Plan to explain RA and its benefits • Pilot procedures • Develop procedures for implementing, controlling and monitoring RA • Hold negotiation meetings • Have a backup plan in case RA fails • Seize naturally occurring opportunities to facilitate RA • Match RA design with experimental context

Chapter 10 - Treatment Implementation & Attrition • The problems of treatment implementation and attrition threaten the very reason for doing an experiment: to get a good estimate of a treatment effect.

Treatment Implementation • Failure to get the full intervention • Compliance • Assignment • Crossing over to get a different treatment • Treatment diffusion

Inducing & Measuring Implementation • Induction (implemented as intended) • Components of implementation • Treatment delivery • Treatment receipt • Treatment adherence** • Need to increase these three and measure each

Treatment Delivery • How to improve: Treatment manuals, training service providers, giving verbal reminders to providers to include all treatment procedures, on-the-spot instructions to them during treatments, administering treatment by videotape/audiotape. • Complex, burdensome, long, inconvenience, expensive treatments that require recipient to alter lifestyle will be delivered with less integrity • Measured withstaff meetings, reviewing/scoring tapes, assess differential delivery.

Treatment Receipt • How to improve: Written handouts, using established communication strategies (i.e. repetition, making deliverer appear expert, question recipient about key treatment features to induce cognitive processing, have recipients keep logs of treatment-related activities) • Measured using manipulation checks, written tests of change in recipients’ experience during treatment, monitoring physiological changes that the treatment should induce or asking recipient if they are confident in applying treatment skills.

Treatment Adherence • Threats: Lack time, forget to do it, unsure of correct treatment procedures, disappointed by initial results, lack access to appropriate setting, lose motivation. • How to improve: Assigning written homework, using family members to encourage adherence, physical aids, motivational cards, reinforcements • Measured by interviewing recipients and other informants, biological assays

Overlooked Targets for Implementation Assessments • Extra-study treatments that participants are getting while they are in an experiment. • Those who are assigned to a no-treatment control condition. • Unplanned things that service providers do in treatment (need capacity for discovery - Qualitative methods)

Assessing Program Models • Implementation involves inputs that the treatment requires, contextual issues, funding. • Anticipate potential breakdowns in intervention • Descriptions of the context of implementation • 2 methods to accomplish these goals: • Process model • Good description of all these matters in study reports

Treatment Implementation in Efficacy and Effectiveness Studies • Efficacy trials: treatments are often standardized and full implementation is the goal (usually prefered when treatment is first being studied) • Effectiveness trials: inclusion criteria loosened and recipient compliance may be left variable so that researchers can gauge how well it will perform in less-than-ideal circumstances - yields an internally valid estimate of the effectiveness of that treatment-as-standardized-and-implemented.

Analyses Taking Implementation into Account • When treatment implementation data are available, experiments may analyze them in three ways: • An intent-to-treat analysis • An analysis by amount of treatment actually received • By one of a variety of newly-developed analyses that try to combine some of the benefits of the first two options.

POST-ASSIGNMENT ATTRITION • Post-assignment attrition: any loss of response from participants that occurs after participants are randomly assigned to conditions. • Lowers statistical power • Treatment-correlated attrition threatens internal validity in randomized experiment

Preventing Attrition • Attrition caused by treatment or by the research procedures • Premature termination • Noncompliance with medication • Interpersonal conflict between research staff and participants • Solutions • (treatment) Manipulation - informing participants of the nature of the treatment and the expectations a client should have and tailoring treatments to more closely match client expectations. • (research process) Debriefing participants and asking dropouts why they failed to return

Retention & Tracking Strategies • Gather complete location information at baseline from the participants, friends,or relatives and any available records or agencies that may know their whereabouts • Establish formal and informal relationships with public and private agencies that may help find participants • Create a project identity • Emphasize the importance of tracking to project staff and ensure they are well-supported and compensated

Retention and Tracking Strategies cont. • Use the simplest and cheapest tracking methods first • Make research involvement convenient and rewarding for participants • Expend the greatest amount of tracking effort at the initial follow-up period when most attrition occurs • Customize tracking efforts to the individual participant’s situation and the study’s circumstance.

Preventing Treatment Attrition versus Measurement Attrition • Measurement attrition: a failure to complete outcome measurement, whether or not treatment is completed • Treatment attrition: those research participants who do not continue in treatment, whether or not they continue taking the measurement protocol. • Prevent measurement attrition even when you cannot prevent treatment attrition

Minimizing Time and Obstacles Between Randomization & Treatment • Attrition is lower when the time and obstacles between random assignment and treatment implementation are minimized • “Running-in” procedure • Methods to minimize attrition - increase selectivity, reduce generalizability

Minimizing Treatment-Correlated Attrition • Differential attrition is more important than total attrition as a threat to internal validity. • Can result from various factors: • Differential vigilance • Desirability • Solutions • Informed-consent procedure - agrees to accept assignment to any experimental condition - reduces generalizability • Two-stage informed-consent procedure • Request participants cooperation with measurement - assignment to conditions is made from those who consent • Request agreement to the experimental conditions from those participants assigned to a treatment (not control unless there areethical issues) • Those who refuse second consent are continued in the measurement protocol to which they already consented, reducing measurement attrition

Preventing Measurement Attrition • Use of personal or telephone (versus) mail surveys • Use of incentives to answer • Providing prior notice of the questionnaire’s arrival • Using the foot-in-the-door method that gets the respondent to agree to a smaller task first and a larger one later - ethics?? • Personalizing letters and other forms of contact • Follow-up letters • Appealing so social values/flattery were not effective

A flawed approach - replacing Dropouts • Would only solve attrition as an internal validity threat if: • Both attrition and replacement are random • Both former and replacement participants have the same latent characteristics, especially as pertains to outcome

Analyses of Attrition Goal: how much it threatens the validity of a conclusion abouttreatment effectiveness • Simple Descriptive Analyses • Identifying Different Patterns of Attrition • Accounting for Attrition when Estimating Effects • Imputing Values for Missing Data • Estimating Effects in Data Sets with Attrition

Exercise • Same groups as before • Decide on a research topic of interest to you • Discuss how you would design your experiment to maximize valid treatment implementation and minimize attrition. 10 minutes

Practical Problems