1 / 19

Tanja Burgard, ZPID Nadine Kasten, University of Trier Michael Bosnjak, ZPID

Response rates in psychological online surveys. A meta-analysis on the effects of study design and time. Tanja Burgard, ZPID Nadine Kasten, University of Trier Michael Bosnjak, ZPID. Talk at the Conference of the European Survey Research Association (ESRA)

Pat_Xavi
Download Presentation

Tanja Burgard, ZPID Nadine Kasten, University of Trier Michael Bosnjak, ZPID

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Response rates in psychological online surveys. A meta-analysis on the effects of study design and time. Tanja Burgard, ZPID Nadine Kasten, University of Trier Michael Bosnjak, ZPID Talk at the Conference ofthe European Survey Research Association (ESRA) 16.07.2019, Zagreb (Croatia)

  2. PROBLEM: DECLINING PARTICIPATION IN PSYCHOLOGICAL SURVEYS Response Decreasedwillingness due tooversurveying • Nonresponsehasincreasedsincethe 1990s • in socialsciencesandpolitics (Brick & Williams, 2013) and • in counselingandclinicalpsychology (Van Horn & al., 2009) • Problem: OftensystematicBiasedresults • Trend in recentyears: Growingpopularityof online surveys in psychology • Web surveysyieldlowerresponseratesthanothersurveymodes (LozarManfreda& al., 2008) • Web surveysmaybelessrepresentative • Dynamic field: Growingnumberofinternetusersandincrease in web surveysChange over time? Time Response Increasedaccessandacceptanceoftheinternet Time

  3. RESEARCH QUESTIONS • Howhasthewillingnesstoparticipate in psychological online surveysdevelopedovertime? • Weassumedeclining initial responserates. • Whichfurther variables do moderate initial responserates? • Weexploretheinfluenceofthefollowingmoderators: Mode ofinvitation, incentivesandlengthofthequestionnaire • The findingsshouldguideresearchers in howtooptimallyimplementpsychological online surveysyielding high responserates • Wefocus on samples with a relevant share of people suffering from depressionoranxietydisorder  epidemiologically relevant subgroup in psychology

  4. HYPOTHESES ON MODERATORS OF PARTICIPATION • H 1: The initial response rate in psychological online surveyshasdecreasedover time. • Amountofcommunicationhasincreased, aswellasnumberofsurveysandscientificstudies, especiallyonline surveysbecomemorepopularoversurveying • More informationhastobeprocessed, lessstimulationandattentiontosinglecommunicationrequests H 2: The higherthenumberofitems in a questionnaire, thelowertheinitialresponse rate. H 3: Initial responseratesarehigher, ifincentivesaregivenforparticipation. Cultural changetoindividualism: Respondentsfeellesssociallyobligedandratherbasetheirdecisiontoparticipateon costsandbenefits(Johnson & al., 2002). H 4: The morepersonal thecontactmodeoftheinvitation, thehigherthe initial responserate. In themassofcommunicationrequests, personal addressis a waytogetattention.

  5. ELIGIBILITY CRITERIA • Population: At least 30 % of adults (> 18 years) withdepressionoranxietydisorder, no student samples • Outcomes: Response rate or information on the flow of participants that allow the computation of the response rate • Study type: Psychological online surveys reporting on relevant study design characteristics (burden of participation or incentives or contact mode)

  6. LITERATURE SEARCH: DATABASES AND SEARCH TERMS • 10 databasessearched: PsycInfo, Embase, Medline & In-ProcessCitations, MedlineAheadof Print & Daily Update, Campbell Library, Science Citation Index, SocIndex, CENTRAL, PubPsych, ReStore • Conference proceedingssearchedmanually: ESRA and AAPOR (thepastthreeyears) • Search terms: ("participation rate" OR "response rate“) AND ("online survey" OR "online surveys" OR "internetsurvey" OR "electronic survey" OR "email survey") AND (Anxi* OR depress*)

  7. LITERATURE SELECTION PROCESS: PRISMA FLOW

  8. CODING AND DATA EXTRACTION

  9. ANALYSIS METHOD • All analysesareconductedusingthemetaforpackage (2.1-0)in R (Viechtbauer 2010) • Wecomputedfourmeta-analyticmodels: • Overall effect: Random-Effectsmodelwithoutmoderators • Time effect: Random-Effectsmodelwithonlypublicationyearasmoderator • Effectofstudycharacteristics: Random-Effectsmodelwithall hypothesizedmoderatorvariables (publicationyear, modeofinvitation, lengthsurvey, incentives) • Fullmodel + controls: Model 3 + additionallycontrollingfortype ofrecruitment, useoffundsforstudyconductionandmeanageofthe sample

  10. OVERALL EFFECT Cumulative forest plot Random-Effects Model (k = 20) tau^2: 0.063 I^2: 99.92% Test forHeterogeneity:p-val< .001 Overall mean: 0.428 [0.317; 0.539]

  11. TIME EFFECT Random-Effects Model (k = 20) Estimatedeffectofpublicationyear on RR: • -0.112, p-val < .034 • 15.75 % ofheterogeneityareaccountedforbyPubyear

  12. STUDY DESIGN EFFECTS • The hypothesizedstudy design effects (publicationyear, numberofitems, useofincentivesandmodeofinvitation) explain 29 % oftheheterogeneityofthedependent variable RR • Ifadditionallycontrolledforthe type ofrecruitment, theuseoffundsandthemeanageofthe sample, 53 % oftheheterogeneityareexplained • All in all, theresultsofthe meta-regression supportthehypothesesfortheeffectsofpublicationyear (H1), numberof item (H2) andthemodeofinvitation (H4) • The hypothesizedeffectofincentives (H3) cannotbesupported

  13. DISCUSSION AND OUTLOOK • Hypothesizedinfluences on participationmostlyconfirmed: • The meanresponse rate isabout 43 % • Response ratesarelower in morerecentyearsand in caseoflongerquestionnaires • More personal formsofapproachingpeopleyieldhigherresponseratesthan E-Mail invitations • Effectofincentivescould not beconfirmed(perhaps not enoughevidencetodrawconclusions, due tosmallnumberofstudiesthatreporteduseofincentives) • Further evidenceneeded! • Meta-analysis restrictedtopopulationswithdepressionoranxietydisordersvalidationwithotherpopulationsimportant! • Weneedmoreevidencefrom online surveysusingincentivestotestthiseffect, too

  14. LITERATURE Brick; Williams (2013): Explainingrisingnonresponserates in cross-sectionalsurveys. Annalsofthe American Academy ofpoliticalandsocialscience, 645(1), 36-59. Cook; Heath; Thompson (2000): A meta-analysis ofresponserates in web- or internet-basedsurveys. Educational andpsychologicalmeasurement, 60(6), 821-836. Johnson; O‘Rourke; Burris; Owens (2002): Culture and Survey Nonresponse. In: Groves; Dillman; Eltinge; Little (Hrsg.): Survey Nonresponse. John Wiley & Sons, New York. Lipsey; Wilson (2001): Practical Meta-Analysis. Sage Publications, Thousand Oaks. LozarManfreda; Bosnjak; Berzelak; Haas; Vehovar (2008): Web surveys versus othersurveymodes: A meta-analysis comparingresponserates. Journal ofthemarketresearchsociety, 50(1), 79-104. Shi; Fan (2008): Comparingresponseratesfrom web and mail surveys. Field methods, 20(3), 249-271. The American Associationfor Public Opinion Research (2016). Standard Definitions: Final Dispositionsof Case Codes and Outcome Rates for Surveys. 9th edition. AAPOR Tourangeau; Rips; Rasinski, (2000): The psychologyofsurveyresponse. Cambridge University Press. Van Horn; Green; Martinussen (2009): Survey Response Rates and Survey Administration in Counselingand Clinical Psychology. Educational andpsychologicalmeasurement, 69(3), 389-403. Viechtbauer(2010): Conducting Meta-Analyses in R withthemetafor Package. Journal of Statistical Software, 36(3), 1-48.

  15. DEFINING AND COMPUTING RESPONSE RATES AS MAIN OUTCOME • Information retrievedfromstudies (AAPOR 2016) • A: Participantsprovidingsufficientinformation • B: Eligible, but non-interview (Refusal, nocontact, languagebarrier) • C: Unknowneligibility, non-interview (Nothingreturned, undelivered) • D: Not eligible (screened out, quotafilled) • Computationof Response Rate (RR) • RR = Numberofrespondersprovidingusableresponse (A) / numberofeligible sample units (A+B+C) • Problems: • Missinginformation on flowofparticipants • Online surveysoftenself-selected, information on populationtargeted / reachedandrefusals not availablenoresponseratescomputable!

  16. ELIGIBILITY CRITERIA

  17. OVERVIEW: VARIABLE DISTRIBUTION AND CORRELATION MATRIX

  18. META-ANALYTIC MIXED-EFFECTS MODELS

  19. MODEL PREDICTIONS FOR RESPONSE RATES

More Related