in Online Research

1. Sigma Validation Ensuring Respondent Quality in Online Research

2. Who is Sigma? • Sigma Validation created the Duplicate Number® Search in 1978 for General Foods, now part of Kraft. • In addition to Kraft, many consumer product manufacturers and research companies require the use of Duplicate Number® Validation on all of their consumer research studies. • Duplicate Email™ Search is Sigma Validation’s latest innovation for validating online research.

3. Why Validate Internet Research?

4. Why Validate Internet Research? • Professional Respondents • These consumers find any way they can to qualify for your study – and for the offered-incentive. • For instance, you may expect that your respondents are females, ages 25-54. However, the same email address is registered through a different panel company as a male, age 18-24.

5. Why Validate Internet Research? • Respondent Overlap • You may be speaking with the same group of respondents over and over again, through the same or different panel companies. • If the overlap is significant, your results may be skewed.

6. Why Validate Internet Research? • Screening Requirements Not Being Adhered To • Part of your screening requirements might be that a respondent did not participate in another study for a specific product category within a certain time period. • However, when including studies completed by other panel companies, a significant percentage of your respondents might not have qualified for your study based on your specifications.

7. Why Validate Internet Research? • Based on a recent study by ComScore, “more than 30% of all online surveys are completed by less than 0.25% of the population.” • “This small group takes an average of 80 surveys over a 90 day period” • “Members of the panels offered by most of the leading online survey suppliers are, on average, members of as many as seven other panels.” • ComScore uses observation-based research to track their members, and to identify these over-active respondents within their panel. Source: ComScore Website: http://www.comscore.com/custom-research/sample.asp.

8. Why Validate Internet Research? • According to an article in the July 2005 edition of the MRA’s Alert! Magazine, Larry Fisher of Synovate stated that • “Research indicates that some 20-23% of US adults account for 100% of survey respondents in a given year, regardless of whether random sample or panel based.”

9. Why Validate Internet Research? • Based on a White Paper by Rachael Krupek of 20/20 Research-Online: • “Almost all paid survey sites encourage members to join all 250-450+ research companies’ panels on their list so their members can have the chance to participate in hundreds of surveys, focus groups and mystery shops.” • “Some sites even offer software to ‘help you fill out your surveys up to 300% faster.’ In essence, these sites teach people how to be professional respondents.” Source: September 1, 2004 White Paper entitled “Handling Paid Survey Sites.” http://www.qualtalk.com/news/wp040901.htm

10. Why Validate Internet Research? • In Harvey Lauer’s “You say evolution, I say devolution” article that appeared in the July/August 2005 issue of Quirk’s Marketing Research Review, he asserts: • “Technology and economic incentives may be transforming a pool of once-diligent, civic-minded respondents into a horde of game-playing prize mongers who view survey content as a necessary evil – an annoying obstacle to a grand prize, to be dispatched as quickly as possible.”

11. Why Validate Internet Research? • One panel company said they currently identify over 500 respondents a week trying to join their panel using the same or different email addresses, slight variations in the physical address and other techniques • What is your panel company doing to eliminate respondent duplication within their own panel?

12. Why Validate Internet Research? • A research end-user added a question to their online survey and found that “on average, respondents say that they participate in 5-7 online studies per month.” • Do respondents accurately estimate self-reported participation? • What is an “acceptable” number of studies per month? Source: Client of Sigma Validation, June 2005

13. Why Validate Internet Research? • In cases where the incidence for a study is low, panel companies tend to supplement their sample with respondents from other panels. • What is your panel company doing to eliminate respondent duplication across panel lists?

14. How to ensure Respondent Quality?

15. Taming The Wild, Wild West

16. How To Ensure Respondent Quality? • Ask questions of your panel company • How is the panel recruited? • How large is the “active” panel? • How many new respondents are recruited each month?

17. How To Ensure Respondent Quality? • Ask questions of your panel company • What panel management procedures are in place? • How often do they invite/allow respondents to participate? • How do they identify and eliminate professional respondents within their database? • If they supplement their panel for low-incidence studies with respondents from other panels, how do they eliminate duplication?

18. How To Ensure Respondent Quality? • Ask questions of your panel company • What cleaning processes are used after data collection? • Do they check for duplicate respondents by IP address, physical address, unusual domain names, etc.? • Do they check the data for peculiarities in the responses? • Do they check the survey time from start to completion; and the time between questions?

19. How To Ensure Respondent Quality? • “Inspect” your online survey provider • Request the details (or a physical/virtual tour) of your online survey provider’s program for panel management & data-cleaning. • Set up a procedure to randomly audit surveys conducted by your online sample providers

20. How To Ensure Respondent Quality? • Implement online research standards • Set internal standards for what you expect of your online research provider • Randomly audit online providers for their compliance to your standards. • Modify your standards, as necessary, based on changes in the industry.

21. Expect Then Inspect

22. Expect then Inspect • EXPECT that the provider of your online research sample is: • Tracking the participation frequency of their panel members. • Using state-of-the-art techniques for identifying and eliminating professional respondents from within their panel and after data collection. • Screening respondents based on your specified requirements.

23. Expect then Inspect • EXPECT that the respondents in your study are • Answering your screening questions accurately. • Participating in your studies because they qualify, and not only for the offered-incentive. • Not the same people who responded to your last survey, conducted through the same or a different panel company.

24. Expect then Inspect • INSPECT • Ask specific questions of your panel company to ensure appropriate procedures are in place to identify and remove professional and duplicate respondents; and suspicious survey data. • Use some type of validation or auditing method to ensure that all of these expectations are being met.

25. In Summary • There are many reasons for validating online research including • the identification and removal of suspicious data • ensuring that your online sample providers have instituted basic best practices before and after data collection • Research professionals can help ensure the respondent-level quality of their studies • by asking specific questions of their panel companies and online sample providers to ensure best practices are in place • by auditing random surveys and the best practices of their online sample providers • by setting internal research standards, and modifying them based on new information or changes in the industry

26. Thank you! Mary Beth Weber MBWeber@SigmaValidation.com 1275 15th Street Fort Lee, NJ 07024 Tel: 201-886-0550  Fax: 201-886-1413 www.SigmaValidation.com