1 / 43

Successful Researchers spend Time Constructing VALID Indicators

As the most commonly used indicator is a survey/questionnaire, the following power point provides detailed instructions for construction of valid indicators using this data collection technique . Why is it important to construct valid indicators? Garbage in – Garbage OUT!

michael-warren
Download Presentation

Successful Researchers spend Time Constructing VALID Indicators

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. As the most commonly used indicator is a survey/questionnaire, the following power point provides detailed instructions for construction of valid indicators using this data collection technique. • Why is it important to construct valid indicators? • Garbage in – Garbage OUT! • Your indicator may appear professional, but if it is not valid, you have garbage. • You may have a large sample size, but if your indicator is not valid, you have garbage. • You may use “high powered” statistics, but if your indicator is not valid, you have garbage. • You may have a great program, but if your indicators are not valid, then your evaluation of the program is garbage. • You may have a well-written journal article, but if your indicator is not valid, you have garbage. Successful Researchers spend Time Constructing VALID Indicators

  2. Step 1 – Make Decisions about TYPES of Indicators • Level of measurement • Single vs. multiple indicators • Closed vs. open-ended indicators • Step 2 – Consider Content of Questions • Make it easy for respondents • Make it easy for those who input data • Construct indicators that yield reliable (consistent) and valid (accurate) results • Step 3 – Consider Format of Survey/Questionnaire • Step 4 – Know How to Evaluate Surveys • Step 5 – Construct a Codebook Steps to Constructing a Survey Instrument

  3. Level of measurement • Measure at highest level possible. Determined by • How precise can respondents be? • For example - Can they remember their exact income? • How precise are they willing to be? • For example - Will they tell you how many STIs they have? • Single vs. multiple indicators • Multiple indicators should be used to measure different dimensions of same concept or ask about multidimensional concept. • Examples • Scales used to measure marital happiness, depression, college readiness (i.e., SAT, ACT), etc. • Questions used to measure different dimensions of social class (e.g., income, occupation, education) Step 1 – Make Decisions about Types of Indicators

  4. Close vs. open-ended questions • Two types of closed ended questions 1. Unordered - Ask about type of activity Example - Which of the following activities do you engage in before going to bed? • Reading • Eating • Both of the above 2. Ordered - Ask about degree or strength of reaction to statement Example -Indicate how satisfied you are with the following: VD DS S VS USU Extension Services 1 2 3 4 County Extension Agent 1 2 3 4 Step 1 – Make Decisions about Types of Indicators

  5. Close vs. open-ended • Open-ended questions • What are the advantages? • Important for exploratory research . For instance, use when you do not yet know WHAT questions to ask. • Provide more in-depth information about topic. • What are the disadvantages? • Time consuming. Coding is required once data is collected • Validity and reliability are dependent on the coders and their skills. • Examples of open-ended questions commonly used in evaluation research • What are the strengths of this program? • How would you improve this program? • What things have you learned from this presentation that you can share with others? Step 1 – Make Decisions about Types of Indicators

  6. Make it easy for respondents • Ask questions at the 6th grade level – 20 words or less • Avoid the word NOT in negatively worded questions (i.e., I do NOT study every night.) • Avoid Acronyms (i.e., USU, ARD) • Avoid putting blanks in the middle (i.e., I read for pleasure_____ times per week.) • Avoid having respondent rank items (i.e., rank ten most serious crimes from high to low.) • When using contingency questions mark them clearly with arrows • Match questions/statements with respondents in terms of • Interests – Don’t ask questions about something in which they have NO interest. • Competency - Don’t ask questions about things they know nothing about. • Relevance - Don’t ask questions about things that aren’t relevant to them and/or their situation. Step 2 – Consider Content of Questions

  7. Making it easy for respondent • Write at 6th grade reading level – 20 words or less • Instead of • Based on your annual net income for the previous decade, excluding any atypical pay periods or wages accrued by minors living in your household, what would be your best approximate estimate of your socio-economic status? • A better alternative might be • What is your social class? • Avoid the word NOT in negatively worded questions • I very seldom can’t stop worrying about my grades. • A better alternative would be • “ I often worry about my grades.” • This professor is not a competent teacher. • A better alternative would be • This professor is an incompetent teacher • Avoid Acronyms • Do you think CEOs who graduated from USU should attend ARDs if their child has ADD or an STI? • Avoid putting blanks in the middle • I spend ____hours studying for exams. • A better alternative would be • On average, how many hours do you spend studying for exams during a typical week? ____________ hours Step 2 – Examples of Suggestions on Slide 5

  8. Making it easy for respondent • Avoid having respondent rank items • Rank the following leisure activities from 1 to 8 based on how much you enjoy them. __Read books __E-mail a friend __Ride a bike __Eat potato chips __Swim __Talk on my cell phone __Watch T.V. __ Gossip • A better alternative might be Please indicate your two most favorite leisure time activities (1) ________, (2)________ • Use contingency questions marked clearly with arrows. • Match questions/statements with respondents • Interests – Don’t ask teenagers about flat taxes or retirement pensions. • Competency – Don’t ask the yearly income of their great Uncle Fred. • Relevance – Don’t ask singles about their marital happiness. Step 2 – Examples of Suggestions on Slide 5

  9. Make it easy for those who code, input and /or interpret data. You cannot input letters or boxes into the computer. The computer only recognizes numbers. • THEREFORE, have respondents circle numbers for their answers. Avoid letters and boxes. • For answer categories make certain HIGH numbers correspond to POSITIVE answers. • When doing analysis, it is much easier to interpret a number if a high number corresponds to a positive answer. That is the way we are trained to think about numbers. • Use open-ended questions only when necessary. • When possible use a series of open-ended questions to get more precise answers. Step 2 – Consider Content of Questions

  10. Making it easy for you • Have respondent circle numbers for answer categories. Do you grow corn? ____ Yes _____No A better alternative would be Do you grow corn? (1) Yes (2) No • Have high numbers = positive answers What grade did you receive in this class last semester? (1) A (3) C (5) F (2) B (4) D A better alternative would be What grade did you received in this class last semester? (1) F (3) C (5) A (2) D (4) B Please indicate how satisfied you are with the information you received from this presenter. Possible answers range from (1) very satisfied to (4) very dissatisfied. VS S DS VD 1 2 3 4 A better alternative would be Please indicate how satisfied you are with the information you received from this presenter. Possible answers range from (1) very dissatisfied to (4) very satisfied. VD DS S VS 1 2 3 4 Step 2 – Examples of Suggestions On Slide 8

  11. Making it easy for you • When possible, use a series of open- ended questions to get more precise answers. This can make it much easier to code and interpret your findings. • What do you do for a living? ________ • A better alternative might be to ask a series of questions • Where do you work? ____________________ • If you work for a company or organization, what is the name of that company/organization? _____________ • In a typical day, what types of tasks do you complete?____________________________ Step 2 – Examples of Suggestions On Slide 8

  12. Increase Validity/Reliability • Wording of Question • Be as Precise/specific as possible • When asking about frequency, ask for a specified time • (i.e., On average, how often, per week do you telephone your mother? ___) • When asking for a number, ask for as precise a number as possible • (i.e., How much time do you spend watching TV during an average day? ____ hours ____ minutes) • When asking for a number, try to be exact • (i.e., How old were you on your last birthday?) Step 2 – Consider Content of Questions

  13. Increasing Validity/Reliability • Wording of Question • Avoid Slang – “hot” “cool” • Do you think it is cool to skip class? • A better alternative might be • Do you think students approve of skipping classes? • Avoid Bias or Leading Questions • Cancer is a devastating illness that kills millions of Americans every year. How much money do you think we should spend finding a cure for this illness? • A better alternative might be How much money do you contribute to cancer research? ________ dollars per year. • Women who are raped never recover. They only survive. What penalty do you think rapists should receive? • A better alternative might be • Indicate the extent to which you agree with the following statements: • Current penalties for convicted rapists are just. • Penalties for convicted rapists should be more severe. • Penalties for convicted rapists are too harsh. Step 2 – Consider Content of Questions

  14. Increasing Validity/Reliability • Wording of Question • For Indices • Use statements – not questions SD D NAND A SA Do you enjoy the firework 1 2 3 4 5 at the county fair? A better alternative would be SD D NAND A SA I enjoy the fireworks 1 2 3 4 5 at the county fair. Step 2 – Consider Content of Questions

  15. Increasing Validity/Reliability • Wording of Question • For Indices • Avoid response set SD D A SA I loved this program. 1 2 3 4 The teachers were very friendly 1 2 3 4 I learned a great deal. 1 2 3 4 I want to do this again 1 2 3 4 A better alternative would be SD D A SA I loved this program 1 2 3 4 The teachers were unfriendly 1 2 3 4 I learned nothing new 1 2 3 4 I want to do this again 1 2 3 4 Step 2 – Consider Content of Questions

  16. Increasing Validity/Reliability • Wording of Question • For Indices • Avoid using NOT in a negative statement SD D A SA I did NOT like the program 1 2 3 4 The teachers were Not friendly 1 2 3 4 I did NOT learn anything 1 2 3 4 A better alternative would be SD D A SA I disliked this program 1 2 3 4 The teachers were unfriendly 1 2 3 4 I learned nothing new 1 2 3 4 Step 2 – Consider Content of Questions

  17. Increasing Validity/Reliability • Wording of Questions • For indices • Use multiple statements to measure multidimensional concepts (i.e., social distance) SD D NADA A SA I would live with an Asian 1 2 3 4 5 I would go to parties with Asians 1 2 3 4 5 I would feel uncomfortable sharing an office with an African American 1 2 3 4 5 My family would disapprove if 1 2 3 4 5 If I married a Hispanic Step 2 – Consider Content of Questions

  18. Increasing Validity/Reliability • Wording of Questions • For Indices • Avoid double barreled/double blind statements SD D A SA Men and women are poor communicators 1 2 3 4 I like the staff and the 1 2 3 4 teachers My health has changed 1 2 3 4 since last year • A better alternative might be SD D A SA Men are poor communicators 1 2 3 4 Women are poor communicators 1 2 3 4 I liked the staff . 1 2 3 4 I liked the teachers 1 2 3 4 My health is better 1 2 3 4 My health is worse 1 2 3 4 Step 2 – Consider Content of Questions

  19. Wording of Answer Categories • Exhaustive • Cover all possible answer Word questions so all answers are possible. Example How often, IF EVER, do you attend football games? ____ times per semester. For instance Use the answer category “Other (specify)______” whenever it is appropriate. Word answer categories so all possible answers are acceptable. For instance, Hitler was a 1. Very negative role model 2. Negative role model A better alternative would be Hitler was a 1. Very negative role model 3. Positive role model 2. Negative role model 4. Very positive role model Step 2 – Consider Content of Questions

  20. Wording of Answer Categories • Use ALWAYS and NEVER with great caution Please indicate the extent to which you do the following. Possible responses are (1) Never, (2) Seldom, (3) Sometimes, (4) Often, and (5) Always N S ST O A I read the newsletter mailed 1 2 3 4 5 out by the extension agent. A better alternative would be to replace NEVER and ALWAYS with ALMOST NEVER and ALMOST ALWAYS. AN S ST O AA I read the newsletter mailed 1 2 3 4 5 out by the extension agent. Step 2 – Consider Content of Questions

  21. Wording of Answer Categories • Mutually exclusive • NEVER have two possible right answers • For instance - • How much money did you make last year? (1) $10 ,000 to $20,000 (2) $20,000 to $40,000 (3) $40,000 to $60,000 • A better alternative would be (1) $10,000 to $19,999 (2) $20,000 to $39,999 (3) $40,000 to $59,999 Step 2 – Consider Content of Questions

  22. Wording of Answer Categories • Avoid offensive/biased answer categories • For instance • Use overweight NOT obese • Use pro-life NOT anti-abortion • Use pro-choice NOT pro-abortion • Use thin NOT skinny • Use government assistance NOT welfare • For instance • What do you do when you don’t go to class? (Circle all that apply) 1. Act Like a Lazy Slob 2. Waste my time A better alternative would be How do you spent your time when you don’t attend class? ________________________ Step 2 – Consider Content of Questions

  23. Wording of Answer Categories • Be as precise/specific as possible • Using YES/NO as answer categories • Disadvantages Doesn’t give you much variation. Is not precise enough. • Advantages Sometimes it is the only plausible answer. • Using “Don’t Know” as an answer category • Disadvantages Sometimes, it is NOT a plausible answer. Can result in response set, as respondents may see it as the easier, though not most accurate answer. • Advantages Can add important variation to possible answers, and REMEMBER you need/ want variation when you start doing analysis. Step 2 – Consider Content of Questions

  24. General Guidelines • Using Numbers • For single items – number from top to bottom in columns. • What grade did you receive? 1. F 4. B 2. D 5. A 3. C • For index : (1) put numbers under symbols, (2) number from left to right (3) number from lower to higher SD D NAD A SA 1 2 3 4 5 1 2 3 4 5 • Professional Appearance • Spacing – even and balanced • Uncluttered appearance • No spelling or grammar errors • Can use shading, etc. Step 3 – Consider Format of Survey

  25. Required Sections • Textbox • Should contain the following information • General statement about the purpose of survey • Statement indicating the time survey will take • Statement that indicates it is anonymous or confidential • Statement that indicates it is voluntary • Contact information for those wanting more information • Basic socio-demographic questions • Questions measuring indicators/variables • Contingency questions • All parts should be on the same page • Indices/Scales • The entire index/scales should be on same page – if possible. If you put it on two pages, then repeat headings. Step 3 – Consider Format of Survey

  26. SectionPlacement Textbox Top of first page of survey Sociodemographic First page of survey Questions measuring indicators Sensitive questions Bottom of last page of survey Independent variable First page of survey Dependent variable Second page of survey Open-ended questions End of survey Contingency Questions Throughout the survey Index (ices) Generally middle or end of survey Placement of Sections

  27. Appearance • Does it look professional? • Is the text “balanced”? • Did you use numbers? Are they in correct order? • Statement at Top • Does it include relevant/complete information? • Is it in a textbox? • Questions • Format – for index or scale • Does it have a textbox at top with instructions? • Is it numbered from left to right – lower to higher? • Did you use statements – not questions? • Are statements logically related to each other? • Wording • Did you use a spelling and grammar check? • Did you avoid – biased statements/word, slang, abbreviations, double-barreled questions/statements, etc.? • Answer categories -measured at highest level possible? -exhaustive/mutually exclusive? • Do you test your hypotheses/measure your indicators? Step 4 - Criteria for Evaluation - Questions You Must ASk

  28. Use numbers not letters • Measure indicators/variables at the highest level possible (i.e., the ratio level, whenever possible) • Ask only relevant questions • Don’t ask college students if they graduated from high school • Use statements rather than questions in indices. You can’t agree with a question! • Avoid bias. If you can predict what answer the researcher “wants” then it is a biased statement or question. • Make wording across answer categories consistent • For instance, If you use pro-life, then use pro-choice • For instance, If you use African American, then use Asian American. Avoid Common Mistakes

  29. You can use the following chart or one like it to evaluate the validity of the indicators used in your survey/questionnaire Concept Instrument Face Validity (Y/N) For each indicator, ask yourself the following question? • Can you match a question or questions on your survey with each indicator you are using to measure impact? • Are you measuring attitudes or behavior? Does the question/statement in the survey match? • If you are measuring a multi-dimensional concept – do you have more than one question/statement? Then determine if your concept has face validity (i.e., it logically appears to measure the concept). Evaluating your Indicators

  30. You can use the following chart or one like it to evaluate the validity of the indicators used in your survey/questionnaire Concept _________ Instrument _________ Face Validity (Y/N) Impact of program How often do you recycle Y on recycling behavior your drink containers? Impact on attitudes toward To what extent do you agree Y water conservation that Utah residents must learn to conserve water or face shortages? For each indicator, ask yourself the following question? • Can you match a question or questions on your survey with each indicator you are using to measure impact? • Are you measuring attitudes or behavior? Does the question/statement in the survey match? • If you are measuring a multi-dimensional concept – do you have more than one question/statement? Then determine if your concept has face validity (i.e., it logically appears to measure the concept). Evaluating your IndicatorsExample

  31. You can use the following chart to evaluate the reliability of each of your indicators. Take each concept, and then rate the indicator on the following characteristics. Generally, the higher the score, the more reliable the indicator. Indicator 1Reliability (+/-) Precise Unbiased Not Double barreled Exhaustive Exclusive Not offensive For each of your indicators, ask yourself if you think your indicators would yield consistent results across time. Answer the following questions: • Are questions precise and unambiguous? • Are there any words that may be interpreted differently by different people? (slang) • Are there biased questions that may influence some people’s answers? • Are any questions double barreled? • Are answers exhaustive? (Did you use “Other – please specify” as an answer category?) • Are answers mutually exclusive? • Are there questions that may offend some? Evaluating Your Indicators

  32. Use the following chart to evaluate the reliability of your indicators. Take each concept, and rate the indicator. Generally, the higher the score, the more reliable the indicator. Indicator 1Reliability How would you rate the presentation you just attended? • Fair • Good • Excellent (+/-) Precise - Unbiased - Not Double barreled + Exhaustive - Exclusive + Not offensive + Only 2 of 6. This indicator needs to be reworded For each of your indicators, ask yourself if you think your indicators would yield consistent results across time. Answer the following questions. • Are questions precise and unambiguous? • Are there any words that may be interpreted differently by different people? (slang) • Are there biased questions that may influence some people’s answers? • Are any questions double barreled? • Are answers exhaustive? (Did you use “Other – please specify” as an answer category?) • Are answers mutually exclusive? • Are there questions that may offend some? Evaluating Your Indicators – An Example

  33. General Guidelines • Identify a name for your codebook. It should start with a letter (not a number), and should contain no slashes, hyphens, etc. It should be about 8 characters long. • Put an identification number on each of your surveys. This identification number should be the first variable in your codebook. • Construct a name to identify each statement/question used in your survey to measure each indicator. Do not use slashes/dashes/spaces. This name will appear on your statistical outputs when analyzing this question/statement. • Compose a BRIEF description for each variable/indicator. • Construct a code that corresponds to each variable/indicator. The code should consist of numbers representing each possible response/answer. Step 5 – Constructing a Codebook

  34. Q # Var# Var Name Description ___ Code_____ 1 ID Identification # 1-100 1 2 Sex Sex of respondent 1=male 2=female 3 Age Age of Respondent 18-99 years 4 Fear “I fear my future 1=strongly disagree has few 2=disagree 3=undecided 4=agree 5=strongly agree 5 Hope I have great same hope for my future Step 5 – Constructing a Codebook Format of Codebook

  35. Following is an Example of a Survey • Please note the following: • There is no textbox at the beginning. • The information often provided in the textbox was provided in a cover letter. • There are a number of different indices • You don’t have to always use Strongly Disagree to Strongly Agree. • Some of the questions indicate that you can circle all that apply • This ensures that possible answer categories are exhaustive. • Some questions have an “other – specify” category. • This insures that possible answer categories are exhaustive. Examples of Surveys

  36. ALWAYS DO • Have others proofread your survey and give you suggestions • Pre-test your survey using respondents similar to those in your sample • Submit your survey to IRB for approval • Let respondents know you are associated with USU • NEVER DO • Send out surveys without IRB approval Important Final Comments

  37. Contact Dr. Carol Albrecht Assessment Specialist USU Extension • Phone – 979-777-2421 • E-mail – carol.albrecht@usu.edu For More Information

More Related