Research for Measurement & Analysis

1. Research for Measurement & Analysis Dr. Jim Mirabella Florida Community College at Jacksonville Director of Institutional Research Professor of Statistics & Marketing Research Florida Sterling Conference 2001

2. Getting Started What is research? • Research is the process of collecting and analyzing information in order to increase our understanding of the phenomenon about which we are interested. Why research? Applications?

3. Value of Research in TQM • Customer is king • Requires considerable measurement • Rate company against competitors • Measure employee attitudes • Monitor company performance against benchmark standards

4. Sample vs. Population Population = collection of ALL possible observations Sample = subset of a population Random Sample • representative of a population • all observations have equal chance of being selected

5. Why Do We Use Samples?

6. 6 Steps for Drawing a Sample • Define the Target Population • Identify the Sampling Frame • Select a Sampling Procedure • Determine the Sample Size • Select a Data Collection Technique • Collect the Data

7. Define the Target Population • not necessarily the same as the population • the group that the sample is truly representative of • who are you really attempting to survey?

8. Identify the Sampling Frame • from what source do you plan to draw the sample? • ideally, the source should be representative of the population • the source should not bias the results

9. Select a Sampling Procedure • Random samples • Simple • Stratified • Systematic • Cluster • Non-random samples • Convenience • Judgment

10. Determining Sample Size According to Dr. George Gallup: “you do not need a large sampling proportion to do a good job if you first stir the pot well.” • If you have two pots of soup on the stove (one large, one small), you don’t need to take more spoonfuls from the larger pot to sample the tastes accurately. If sample size and margin of error are not given, how can readers have confidence in results?

11. Determining Sample Size • When should samples be large? • Serious or costly decisions • Time and resources readily available • When should you permit a small sample? • Few major decisions based on results • Only rough estimates needed • High data collection costs • Time constraints

12. Determining Sample Size • How much error will you allow? • difference between actual population value and results from sample • as tolerable error decreases, sample size increases exponentially • How confident must you be? • likelihood that actual value is within tolerable error range • as confidence level increases, sample size increases Population size is only a factor for small populations (<5,000)Note: 80/100 = 278/1,000 = 385/1,000,000

13. Costs Time required Sample size for a given budget Data quantity per respondent Reaches widespread sample Reaches special locations Interaction with respondents Degree of interviewer bias Severity of non-response bias Presentation of visual stimuli Field worker training required PersonalPhoneMailEmail High Medium Low Low Medium Low High Low Small Medium Large Large High Medium Low Medium No Maybe Yes Yes Yes Maybe No Yes Yes Maybe No No High Medium Low Low Low Low High High Yes No Maybe Yes Yes Yes No No Comparing Data Collection Methods

14. Selecting a Sample • Start with total population • Select sampling frame • sampling frame error • Select sample • random sampling error • Gather responses • Non-response error

15. Sampling Error • Difference between results of sample vs. population • Causes: • Bad luck • Sampling bias = tendency to favor the selection of certain data • Bias of non-response

16. Difficulties in Sampling • Using the wrong sampling frame • Iinclude unwanted units or exclude desired units • Predicting elextions with non-voters • Is the phonebook viable? • Not reaching the individuals selected • whoever is home to answer phone • Getting no response or getting a volunteer response • Consumer Reports depends on voluntary responses • Response rates of mail surveys

17. Disasters in Sampling • Getting a volunteer sample • call in 900 number survey • Most beautiful celebrity • Talk show surveys • Using a convenience sample • Are they ever valid?

18. Total Survey Error • Random Sampling Error • Bias (systematic error) • Respondent error • Non-response • Deliberate falsification • Unconscious misrepresentation • Administrative error • Data processing error • Sample selection error • Interviewer error • Interviewer cheating

19. Failure from Sampling Error • Failure to use statistical sampling methods • Random sample • Representative of population • 900# / 800# / internet surveys • Phone books • “Alf Landon wins by landslide” (Literary Digest)

20. Failure from Sampling Error • Ignoring the bias of non-response • most serious limitation of surveys • respondents not available when called on • respondents refuse to cooperate • results represent extreme views • customer response cards at restaurant tables • What you don’t hear is as important as what you do hear • CA$H = Guilt • Dealing with the bias of nonresponse

21. Failure from Sampling Error • Ignoring the bias of non-response (cont) • A women’s talk show polled its audience and asked, “If married women had it to do over again, would they marry the same man?” • What errors exist? • What results would you expect? • Reporting on non-response • When response rate is low, be suspicious of results • When response rate is unreported, be very suspicious of results • Don’t confuse response rate with sample size

22. Should Research be Conducted? • Time constraints • Availability of data • Nature of the decision • Costs vs. benefits

23. Phases of Research • Define the problem • Plan the research design • Plan the sample • Gather the data • Process & analyze the data • Formulate conclusions

24. Primary vs. Secondary Research • Primary = researcher directly collects data • Secondary = researcher uses data already collected by others (literature review) • Inexpensive / can be secured quickly • Unknown accuracy / may not fit the problem

25. Looking at Data Intelligently • What is the source of the data? • reports generated by different systems • Does the data make sense? • Does tonic make you drunk? • Is the information complete? • Do I have everything I need to draw a conclusion? • Is the arithmetic faulty? • budget crisis (50% decrease vs. 50% increase)

26. Reliability vs. Validity • Reliability • How well a test consistently measures its intent • Can outcomes be replicated? • “stability” • Validity • How well test measures what it is supposed to measure • Content validity = does test measure the intended content • “appropriateness”

27. Beware of Faulty Studies All’s well if it ends well? Product A has twice as much pain reliever as Product B for the same price. • Are they the same pain reliever? • B may be more potent and more effective. • Compare Anacin to Aleve

28. Why do we survey customers and employees? • to gauge ourselves against others • to get feedback and make improvements • to show customers/employees we care about their opinions

29. Points for More Useful Surveys • Surveys raise customers’ expectations • Time is one of customers’ most valuable assets • Customers expect to have their feedback used • Asking the question implies you can and will act on results

30. Points for More Useful Surveys • How you ask the question will determine what you get • Attitude question • Measures agreement or satisfaction • How customers think or feel about something in particular • Knowledge questions • Only one correct answer • Does customer know the specifics about a product/service • Behavior questions • How often / how much / when • Measures frequency of a behavior

31. Points for More Useful Surveys • You have only one chance and maybe 30 minutes • There is no second chance to get missing data • Time is precious and surveys are not productive time for customers • Open-ended questions take more time, so budget accordingly

32. Points for More Useful Surveys • The more time you spend in survey development, the less time you will spend in data analysis and interpretation • Open-ended questions are easier to write, but take longer to analyze • Specific unbiased questions are hard to write, but easy to analyze • “Do you plan to return to our hotel as a result of the check-out process?” (What’s wrong with this question?)

33. Points for More Useful Surveys • Whom you ask is as important as what you ask • Sample should be representative and random • Exit surveys don’t address why employees stay • If you only study the 10% bad apples, your results will infer that all apples are bad • 900 numbers to vote for Miss America

34. Points for More Useful Surveys • Before the data are collected, you should know how you want to analyze and use the data • Begin with the end in mind • What will you do with the information • Decide how reports will look to address critical issues • Then, decide the data needed to get to the desired report • Then, write questions that get the desired data • Finally, arrange questions for minimal bias / maximum response

35. Response Much greater Somewhat greater About the same Somewhat less Much less Order First Last Second Fourth Middle Middle Fourth Second Last First 5% 2% 9% 10% 48% 46% 26% 23% 12% 19% Sources of Response Bias How the order in which the alternatives are listed affects the distribution of replies Compared to a year ago, the amount of time spent watching television by my household is:

36. Response Hiring tradesmen Tradesmen and household members Household members Order First Last Middle Middle Last First 15% 11% 33% 31% 52% 58% Sources of Response Bias How the order in which the alternatives are listed affects the distribution of replies Most home repair or improvement projects completed in my home during the past years have been completed by:

37. Survey Misuse in Election Polls • Wording of questions or statements can drive results • Two November 1997 Election Day polls asked how people would vote for proposal • 1st Poll: The City of Houston shall not discriminate against or grant preferential treatment to any individual or group on the basis of race, sex, ethnicity or national origin in the operation of public employment and public contracting. • 2nd Poll: Shall the Charter of the City of Houston be amended to end the use of preferential treatment (affirmative action) in the operation of the City of Houston employment and contracting?

38. Survey Misuse in Election Polls • Results of two polls: Wording For Against Not sure Other Nondiscrimination 68% 16% 15% 1% Affirmative action 47% 34% 18% 1% • Same political action but different levels of support • “Affirmative action” wording was actually used and was rejected by 55-45 vote

39. Election Polls Tell All • Wording of questions can skew results, and , oh yes, people lie! • Reagan vs. Mondale (Friday night swingers) • Likely vs. Registered voters • Giving extra weight to under-represented groups • Who really stops at an exit poll? • What happened in Florida (was it expected)?

40. How to Be a Pollster • Write a question for a public opinion survey that is likely to produce results in favor of building a nuclear reactor in your hometown. • Write a question for a public opinion survey that is likely to produce results showing most people are against building a nuclear reactor in your hometown. • Write a fair question that attempts to accurately measure public opinion about whether or not to build a nuclear reactor in your hometown.

41. Research Design for Dummies Suppose the local college wants to know if they should start offering classes at midnight. • Decision problem = • Research problem = • Target population = • Sampling frame = • Recommended sampling procedure =

42. Business Card Dr. Jim Mirabella Director Institutional Research 501 W. State St. Jacksonville, FL 32202 Phone: 904.632.5054 Fax: 904.632.3364 Suncom: 864.5054 jmirabel@fccj.org

43. Why Do We Use Samples? • Cost • Time • Inaccessibility of the population • Accuracy • Destruction of the observations