New dimensions of quality in online panels
This presentation is the property of its rightful owner.
Sponsored Links
1 / 29

New Dimensions of Quality in Online Panels PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

New Dimensions of Quality in Online Panels. Jacqueline Lorch Vice President, Global Knowledge Management Survey Sampling International. The World Is Moving On…. Why Does It Matter?. Questionnaire Design. Fieldwork. Interpretation. Sampling. What Are The Key Quality Elements?.

Download Presentation

New Dimensions of Quality in Online Panels

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

New dimensions of quality in online panels

New Dimensions of Quality in Online Panels

Jacqueline LorchVice President, Global Knowledge ManagementSurvey Sampling International

The world is moving on

The World Is Moving On…

Why does it matter

Why Does It Matter?

Questionnaire Design




What are the key quality elements

What Are The Key Quality Elements?

  • A broadly-recruited, representative, well-managed pool of respondents

  • Respondents who respond honestly and conscientiously:

    • What guarantees are there to guard against bad data, i.e. respondent cheating or not concentrating/caring in their responses… ?

  • A well-designed survey instrument

How do respondents fit in

How Do Respondents Fit In?

  • Google

    “Market Research”464 million entries

    “Market Research Respondents” 2 million entries

What is a professional respondent

What Is A ‘Professional Respondent’?

  • Some assumptions:

    • Quality of responses will be lower

    • Motivated to maximize incentives, not by intrinsic interest

    • Do too many surveys – become conditioned

  • But what definition?

    • Someone who gets paid?

Honesty is the issue

Honesty Is The Issue

“I am an IT Director with 7 wives…”

New typology of bad respondents

New Typology Of “Bad” Respondents

  • Hyperactive Respondents

    • Too many surveys, too many panels

  • Fraudulent Respondents

    • Misrepresent themselves

  • Inattentive Respondents

    • Don’t put thought into answers

  • Conditioned Respondents

    • Have learned from past surveys

Smith and Hofma Brown, Harris Interactive

Hyperactive respondents

Hyperactive Respondents

  • Do busy panelists provide bad data?

  • Do the most responsive panelists take the surveys the fastest?

  • Are the fastest surveys the ‘worst’?

Hyperactive respondents1

Hyperactive Respondents

  • US SurveySpot panel, 3 month internal tracker

    • Two groups:

      • SurveySpot only

      • One or more other panels

  • No difference in average time taken by each group

  • Similar answers on motivation for joining:

    • Motivated by chance to influence decisions

      • SurveySpot-only: 50%

      • More than one panel membership: 47%

Fraudulent respondents

Fraudulent Respondents

  • How can you tell if someone is who they say they are?

  • How can you stop multiple panel memberships using different identities?

  • How can you tell if someone is just making up the answers?

Who are you today

Who Are You Today?

Ask me something only I would/should know…..

Catch the cheat

Catch The Cheat…

  • Respondents…

    • Who have 23 different ailments

    • Who report using non-existent brands

    • Whose education doesn’t match their profession

Inattentive respondents

Inattentive Respondents

  • Fatigue leads them to skip questions

  • Don’t pay attention to instructions

  • May be inevitable after a certain length of survey

Inattentive respondents1

Inattentive Respondents

  • Related to interview length

  • 5% baseline of inattention?

Respondents speak out

Respondents Speak Out

  • Repetitive questions “sorry this survey was just too long.”

  • “Sometimes it becomes so repetitive you say, ‘to hell with it, I don’t need this.’”

  • “You think you are about done and the same questions start all over again.”

Inattentive respondents2

Inattentive Respondents

  • Time use > 7300 hours = 4 hours of sleep a night

  • Time use > 8760 hours = no sleep, ever.. more hours than in a year

  • 5% baseline of inattention?

  • A really tough question set?

Respondents speak out1

Respondents Speak Out

  • “…absolutely ridiculous… [questions like] ‘if this

    pizza was a person.’ ”

  • “Why is this bottled water like your favorite pet?”

  • “In filling out this survey it asked when I would buy a new house. I said “never”. The next several questions were regarding my new house and it required an answer…So I quit the survey and didn’t finish it.”

Conditioned respondents

Conditioned Respondents

  • Only give the answers they do because of what they have learned from previous surveys

  • Or change behavior as a result of information from surveys

  • Have no redeeming features

  • But do such people exist…..?

Tns experiment

TNS Experiment

  • 3 groups of respondents

    • High frequency – interviewed 5 times (

    • Medium frequency – interviewed 3 times (1.3.5)

    • Low frequency – interviewed 2 times (1.5)

  • + Control group at wave 5

  • Same questionnaire

  • Survey frequency not yet released by TNS

  • UK, France, Germany

  • n = 1202 (control = 1470)

Product usage

Product Usage

High Low

Product purchase

Product Purchase

High Low

Brand awareness

Brand Awareness

High Low

Conditioned respondents1

Conditioned Respondents

  • Evidence is hard to find

    • Maybe surveys aren’t quite so important to respondents as they are to researchers!

  • Could we be over-reacting and losing good respondents?

Respondents speak out2

Respondents Speak Out

  • “I would like to know why it is that practically every time I give my age I am refused the access to the survey? Is there something wrong with being 75 and in good health, mentally and physically?”

  • Why do I spend 10 minutes answering questions on one of your surveys before I am given message ‘Sorry, you didn't qualify for this survey’.

  • “I have not been able to take several [most] surveys because I work in the grocery industry. I don’t think it is fair. I AM A CONSUMER TOO … It is not like I work for a company that makes or sells one brand.”

Respondents speak out3

Respondents Speak Out

  • “…ability of the human eyes to see fonts of microscopic size. What’s with this survey? The font size was like .002.”

  • “I said yes before; why are you going back and asking me the same question.”

  • “Right to the point”…“if they say it’s 5 minutes, it’s 5 minutes.”

  • “By and large ask sensible questions in a straightforward way.”

What needs to happen

What Needs To Happen?

  • Partnership…common terms and definitions

  • Avoid red herrings

  • We are not in the business of supplying bad panelists

  • Survey design is critical

  • We can’t do it alone

  • Login