Quantitative Methods

1. Quantitative Methods Survey Development Prof. Paul Licker, Ph. D.

2. Agenda • Survey Research • Instrumentation • Stages of Instrumentation • Construct-tion • Generation • Pretesting • Pilot Testing • Item Analysis • Reliability Assessment • Validity Assessment

3. Survey Research • Gathering data about a phenomenon using a statistical approach by observing directly or indirectly some subset of the set of phenomena • Often uses individuals’ recollections or perceptions of events or their evaluations of the events

4. Survey Data Gathering Methodologies • Personal Interviews • Group Interviews • Phone Interviews • Mail-out Questionnaires • Hand-out Questionnaires • Clip-and-Mail Questionnaires • Survey of Secondary Data

5. Survey Methodologies Reasoning The goal of our research is to describe or relate the behaviour, opinions, impressions, knowledge, etc. of people in situations. If we cannot directly observe them, their behaviour, etc., we must rely on their own (“subjective”) observations of themselves. Individuals and their actions and opinions are unique, however. No single self-observation can be generalised to everyone.

6. Survey Methodologies Reasoning-2 In order to describe the general situation, we aggregate a series of measurements, observations or judgments about individual (“subjective” impressions of) events. The “objective” description is the average of all these “subjective” descriptions. We assume all biases “average out” and compensate for individual differences.

7. Friendly, familiar, cheap Relatively quick Useful if survey questions are clear Ideal for Descriptive research Uses common skills Hard to control Samples are often very large Has hidden flaws that can be easily ignored Cannot easily be used to draw causal inferences Retrospective Strengths...and…Weaknesses

8. Example: Questionnaires • Questions are written out • Highly structured • Sent out or handed out to a large group • Response is on paper, to be mailed or handed back • Anonymity, ease of reading and response, and clarity are the key qualities • Can use the web or email for distribution

9. Questionnaire Procedure • Create Sample of Questions • Pretest Questions • Derive Sampling Frame, Draw Sample • Distribute Questionnaires Motivate Response Enable Response • Collect Responses • Code and Record Responses

10. 1. Mail-Out Questionnaires • Determine Sampling Frame • Create Package, with response motivator • Pretest all questions • Expect 25% for public campaign, 50% for in-company campaign response • You must explain non-responses

11. Mail-out Questionnaire Example A sample of 1300 members of the ASM was mailed a questionnaire concerning their careers in IS. The questionnaire was pretested on 20 ASM members and fellow academics. Only 256 replied on the first wave. A second wave of 1300 was sent out and an additional 162 replied. Of these 402 were actually usable. Eighteen responses came in after the research was written up.

12. 2. Hand-Out Questionnaires • Get company permission • Use sampling frame • Distribute through company, collect through company • Expect at least 50% response rate • Watch interaction with company situation in terms of date, time, season • Confidentiality is a BIG concern

13. Hand-out Questionnaire Example Using departmental secretaries, ALL 503 professional and managerial staff at a large financial company were given questionnaires on their perceptions of the role of IS in corporate success. A week later the secretaries attempted to collect the responses; only 478 were collected and returned; of these 353 were useful.

14. 3. Mail-in Questionnaires • Subjects “come across” the questionnaire and are self-motivated to complete it and return it (see also electronic variants) • Suffers from strong self-selection problems: only those who are motivated to return it do so…they may have an axe to grind. • Not considered valid research • Similar to radio “talk” shows.

15. Mail-in Questionnaire Example Readers of the Business Timeswere encouraged to fill out, clip, and return a questionnaire about their desires to leave South Africa. 74% of those who professional people who responded said they would leave if they could, which was widely reported and quoted. In fact, those who aren’t intending to leave would never clip the coupon, giving vastly inflated “positive” responses.

16. Examples of Question Format Survey Questionnaire 5. N (4, 5, 6, 7-) Point Scale 6. Grid 7. Filter: If xxxx skip to 9 8. Distribution of 100% A__ % B__% C__% D__% 1. Fill in the blank __________ 2. Multiple Choice (one/more) First Third Fifth Second Fourth Sixth 3. Rank Order Item 1 ___ Item 2 ___ Item 3 ___ Item 4 ___ 4. This one is ___Yes ___ No People will make errors! No Low Mod Hi V.Hi A B C D E F Various choices for each item Give clear di-rections!

17. Relatively inexpensive Can collect lots of data Easy to do Thought to be quick and cheap Management is relatively simple Data need little interpretation No noverbals????? Questions must be ironclad Who is responding What about non-response? Followup is hard Can turn out to be expensive May require artistry, presentation important Questionnaire Strengths…and…Weaknesses

18. Problems to Watch Out For • Low response rates: must be explained • Incomplete response: must be handled • Lack of Pretesting • Bad questions (ambiguous, meaningless) • Open-ended questions: interpretation • Self-selection • Who is responding

19. Electronic Variants of Questionnaire Surveys • Web (requires programming, artistry) • Discussion Groups (may annoy lots of people) • Electronic Mail (can destroy anonymity) • Can gather information quickly over a wide geographical area • Suffers from self-selection • Has a halo effect (+ or -) on IS issues • Sampling frame is a problem • Easy to do very badly after a lot of effort

20. Questionnaire Diagnostic-1 A mail-out questionnaire is sent to 250 customers of a bank to find out expectations concerning E-commerce. The questionnaire contains 142 fill-in-the-blanks questions about finance, technology, banking, and customer expectations of possible E-commerce ventures. Can you expect some problems?

21. Questionnaire Diagnostic-2 Almost 1000 questionnaires are handed out to microcomputer users (obtained from Computer Services) through secretaries in a government department for research examining user-IT relations. The six-page questionnaires are to be picked up in one week by the secretaries. Only 151 of the questionnaires are returned. Will there be any problems in interpretation of the data?

22. Questionnaire Diagnostic-3 Six companies are simultaneously switching to SAP and you want to get impressions from IT staff concerning the switch, so you send out two questionnaires (during the switch and 12 mo. after) by mail to all IT staff. Mostly open-ended questions, the survey has a response rate of 4%. You determine that respondants don’t differ demographically from non-respondants. What’s the problem here?

23. Instrumentation • The development of valid and reliable instruments to measure a phenomenon. • Instrument consists of items. • Items are stimuli to respondents • Items must correspond individually or in sets to constructs • Respondents must be able to give “valid” responses in a reliable way. • Instrument should be efficient.

24. Stages of Instrumentation • Specify domain of constructs • Generate sample of items • Pretest and Pilot test for readability and content validity • Purify instrument, remove extraneous items • Assess validity (four types) • Assess reliability or internal consistency

25. Constructs An instrument measures constructs The instrument suffers from the limitations of ALL instruments, namely: Jiggle (accuracy) Refinement (precision) Failure (external reliability) Consistency (internal reliability) Ease of use, etc.

26. Instrumentation Instrumentation is the task of building an instrument The instrument must be appropriate for the domain of the construct The instrument must be valid and reliable The instrument must be efficient and productive The instrument must be cost-effective

27. Instrument Use Process What the Instrument says The instrument is a lens through which to view the phenomenon Problem areas use Instrument quality Phenomenon (Human Behaviour or Experience) locus

28. Instruments Measure Proxies Theory Concept/ Construct World of Ideas and Concepts World of Reality and Measurement Actual instantiation of the theoretical concept A variable that is highly correlated (generally causally) with the proxy Surrogate Proxy

29. Measurement Challenges 1. Conceptualising precisely to construct2. Instantiating (operationalising step 1) constructs3. Measuring proxies or determining available surrogates that can be measured reliably4. Demonstrating to others’ satisfaction that steps 1 to 3 can be performed validly Surrogate Proxy Valid, reliable measurements

30. Interpretation Challenges 1. Determining that the reverse correspondence among proxies, surrogates and constructs is logically valid.2. Using that correspondence, interpret corresponding relationships among constructs (i.e., concepts) in the theory3. Reason whether or not the implied relationships among the constructs supports or denies support to your theory Theory Concept/ Construct Valid, reliable measurements

31. Types of Validity Construct Is the instrument actually measuring the construct or events or is it its out- comes actually an artifact of the instrument Predictive Can the instrument dis- tinguish different cases (such as with control variables)? Discriminant Do the items making up the variables correlate strongly within factors? Content/Face Does the instrument adequately cover the content of the con- structs? Are the items repre- sentative of the content? Is the instrument constructed “sen- sibly”? Convergent Do the items “converge” on the constructs?

32. Types of Reliability External Instrument can be applied successful- ly in the physical domain desired Instrument is stable in its usage from use to use (intersubject) Internal All elements of a scale consistently measure the desired construct within the instrument (within subject)

33. Construct-tion • The goal is to construct an instrument consisting of variables that are internally reliable, valid reflections of the constructs and practical to administer (in the sense that valid responses can be collected at affordable cost) • Primary tools are argument, debate, pretesting, pilot testing, correlational analysis

34. Generation of Items • Literature survey • Supervisor • Your own intuitiion • Create a large pool, don’t worry about initial size • State items in comparable way, if building scales

35. PreTesting • Use a panel of judges, experts in the field, people familiar with the culture of informants (=respondents) • Can use other doctoral students, lecturers, industrial sponsors, etc.

36. Pilot Testing • Test for readability • Test for respondability (reliability) • Purpose is to create a smaller set of items by eliminating those that pose problems. • Such problems include jargon, form and format, grammar, ambiguity, multidimensionality (two or more questions in one), cultural no-nos, language level, assumed intelligence, etc.)

37. Readability Procedures 1. Read the instrument yourself. Can it be read? 2. Have your supervisor and committee read the instrument 3. Pilot the instrument on a “captive” group first. Do a protocol analysis on this group. Speak with them as they hear each question and go over its possible meanings. Can they begin to answer the questions? 4. Pilot the instrument on a representative small group of respondents.

38. Item Analysis (Purification) • Make sure that the factors apparent in the instrument are in fact “pure” and do not include extraneous factors that are merely artifacts of the questions themselves. • Get rid of “garbage items” • Use item-scale correlation to calculate Cronbach’s Alpha and eliminate those whose elimination will not lower the Alpha.

39. Assessing Reliability • Reliability is of two types: • Item-scale (internal consistency) • Test-retest (external consistency) • Overall alpha of 0.7 or 0.8, even 0.9 should be found.

40. Assessing Validity • Factor analysis: • Do items load “purely” on one factor only? • Do any items fail to load on any factors? • Rule of thumb is to use factor loading of 0.5 as criterion

41. The End Product Internally consistent An exhaustive series of variables The Instrument Mutually Exclusive

42. The End Product That reflect the underlying constructs of your theory The Instument