ScWk 240 Week 7

1. ScWk 240Week 7 Measurement Part 2, & Sampling October 8, 2012 ScWk 240 Week 7

2. Today’s Agenda • Finish Validity & Reliability • Survey development • Intro to Sampling • Literature review – writing tips and your questions ScWk 240 Sampling

3. Review: Potential for systematic or random error? • People are more alert in the morning than after lunch ____________ • The meaning of “living independently” varies cross-culturally __________ • Male respondents tend to minimize their aggressive tendencies _________ • A 16 y.o. taking a test: “I just circle ‘C’ each time and hope for the best” • The person entering data is not careful enough and mis-codes some of the responses ________ ScWk 240 Week 6

4. Student Report • How is validity related to cultural issues? --Dawei ScWk 240 Sampling

5. Reliability • Concerned with whether or not a particular technique done repeatedly would yield the same result each time • Inter-rater reliability—Two researchers rating the same behavior: do they rate them the same? • Test-retest reliability—Is the response from the first test the same as the response from a re-test? • Alternate forms reliability—Is there the same response from slightly different versions of same measure? • Internal consistency reliability & use of coefficient alpha—Are similar questions (about the same concept) in survey are answered in the same way? ScWk 240 Week 6

6. You‘ll see this in your lit review… • Methods section—measures used: the author will report on reliability testing (if done) using the coefficient alpha • A number that tells you, overall, how well the questions in an instrument measuring the same concepts correlate with each other (internal consistency reliability) • A coefficient alpha is considered good if it is .80 or above • The author will then report “The instrument showed good internal consistency reliability” ScWk 240 Week 6

7. Validity • Concerned with the accuracy of meaning—are you measuring what you originally intended? • Types of validity • Face validity – experts’ understanding of concept • Content validity – Does measure cover all relevant dimensions? • Criterion-related validity – Does measure of concept agree with (or relate with) another related concept (external criterion)? • Construct validity – Does measure agree with or correlate with measures of other theoretically related concepts? ScWk 240 Week 6

8. Validity examples • Validity studies show that a set of questions about alcohol dependence correlates well with the observed daily amount of alcohol consumption. Criterion-related validity • Most experts would agree that children with Attention Deficit Disorder with Hyperactivity have difficulty staying on task. Face validity • Couples who test as having low levels of marital satisfaction communicate less effectively, leading to high levels of conflict and, eventually, divorce. Construct validity • The Beck Depression Scale covers those dimensions of depression that are most supported by evidence. Content validity ScWk 240 Week 6

9. Student reports • What are some strategies for ensuring reliability when developing survey questions? -- Amanda • What are some strategies for ensuring validity when developing survey questions? -- Sarah ScWk 240 Sampling

10. Writing survey questions to maximize validity & reliability • Question written at appropriate level of understanding (age, knowledge level, language spoken) • Avoid “double-barreled” question • “Are you satisfied with services and why?” • Questions are sensitively arranged (emotionally-laden ones later) • Instructions make sense • Scaling of attributes makes sense • Consistent language • Mutually exclusive choices ScWk 240 Sampling

11. What’s wrong with this? How satisfied are you with the services you received? • Very satisfied • Somewhat satisfied • Don’t know • Somewhat dissatisfied • Terrible ScWk 240 Sampling

12. Should I use open- or closed-ended questions? • Student Report: What is the main difference between “open-ended” and “closed-ended” questions? --Selene • Qualitative vs. quantitative survey purposes ScWk 240 Sampling

13. Why sampling? • The problem of quality control on the assembly line • Usually can’t observe (take measurements from) 100% of a population • Making the decision on who or what to observe involves understanding • What population do you want to study? • Whether you want breadth or depth of information ***Discussion: So, how do you decide that? ScWk 240 Sampling

14. First, a few terms to know… • Study population--Your study’s focus • As stated in your problem/research statement, and maybe your hypothesis • You want to generalize your study’s results to this larger population. “Generalize” – to extend the findings of your sample to the larger population • Sampling frame--Those eligible to be selected into your study • Could be your entire study population, or more typically it’s a subset of the study population meeting criteria for inclusion in your study • Often, literally, a list (such as a list of agency clients) • Not every study population comes with a handy sampling frame (e.g. homeless people) • Sample--Those ultimately selected to be in your study ScWk 240 Sampling

15. Study Population Sampling Frame Sample ScWk 240 Sampling

16. Example: “The impact of a parenting class on teen parents’ stress levels” All teen parents Sampling Frame—list of all teen parents in a school From this sample, we want to generalize to the population of all teen parents Sample ScWk 240 Sampling

17. Two main types of sampling: • Probability sampling -- everyone in the study population has an equal chance of selection into your study. Keywords: random assignment • Non-probability sampling -- everyone in the study population does not have an equal chance of selection ***Discussion: what are the pros and cons of both? ScWk 240 Sampling

18. The Logic of Probability Sampling • Overall purpose of probability sampling: to allow the researcher to estimate characteristics (parameters) of the study population from a sample’s characteristics • e.g. “How much does the sample represent the population from which it was drawn?” • Why do we want to “estimate population characteristics”? • Because in most cases we don’t really know the real population characteristics (but we’d like to know!) • So, we estimate a population parameter (e.g. average depression score) of the population, from a sample statistic (average depression score of the sample) • Since research means “never having to say you’re certain”, we won’t be 100% perfect in our estimates, although we can quantify how confidently we feel our sample matches the population. How? Stay tuned for ScWk 242. ScWk 240 Sampling

19. How to do probability sampling • Simple random sample—assigning numbers to potential participants and selecting numbers (people) randomly. Similar to picking them out of a hat • Systematic random sample—choosing every kth person--selecting persons at a predetermined interval • Stratified random sampling--Simple or systematic random sampling with sub-groups • Cluster sampling—a multi-stage procedure of randomly choosing levels of analysis units (e.g. cities, neighborhoods, schools, classrooms) ScWk 240 Sampling

20. Non-probability sampling • Quota sampling—sample chosen based on predefined characteristics of study so that sample will have same proportion of those characteristics as in the study population ***What’s the difference between this and stratified random sampling? • Snowball sampling—for difficult-to-locate populations: finding more sample participants based on recommendations from others in the study • Purposivesampling —selecting sample that is thought to yield the most comprehensive understanding of the study’s topic • Convenience sampling (a.k.a. “availability sampling”)—selects participants simply based on their immediate availability. (Note—participants may also coincidentally comprise a purposive sample, but not necessarily) ScWk 240 Sampling

21. Two families of sampling Probability sampling Non-probability sampling Quota sampling Snowball sampling Purposive sampling Convenience sampling • Simple random sampling • Systematic sampling • Stratified random sampling • Cluster sampling ScWk 240 Sampling

22. What type of sampling is this…? • For an experimental study of cognitive behavioral therapy, every 10th adult with anxiety disorders is selected randomly from a sampling frame • An agency manager walks down the hall and asks staff opinions about the new caseload policy • The same agency manager makes sure to ask both women and men • For a study on standardized testing in schools, researchers randomly select cities in California, then randomly select elementary, middle and high schools in those cities • A social work researcher seeks out Latino women who were victims of domestic violence ScWk 240 Sampling

23. Student reports for next week • Name some strategies to recruit difficult-to-reach populations, and retain them for follow up Annalisse • What is required in order to prove causality? Jenna • What is the most important difference between experimental group designs and all others? (Hint—has to do with group assignment) Ashley ScWk 240 Sampling

24. Concepts to know • Closed-ended questions • Open-ended questions • Population • Sample • Sampling frame • Population parameter • Statistic • Generalizibilty • Probability sampling (types) • Non-probability sampling (types) ScWk 240 Sampling