social welfare 580 advanced research methods design l.
Skip this Video
Loading SlideShow in 5 Seconds..
Social Welfare 580 Advanced Research Methods & Design PowerPoint Presentation
Download Presentation
Social Welfare 580 Advanced Research Methods & Design

Loading in 2 Seconds...

play fullscreen
1 / 47

Social Welfare 580 Advanced Research Methods & Design - PowerPoint PPT Presentation

  • Uploaded on

Social Welfare 580 Advanced Research Methods & Design. Measurement Instrument Design Psychometrics. Dr. Caprice Hollins, the director of equity and race relations for Seattle Public Schools.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Social Welfare 580 Advanced Research Methods & Design' - palani

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
social welfare 580 advanced research methods design
Social Welfare 580Advanced Research Methods & Design

Measurement Instrument Design Psychometrics

Dr. Caprice Hollins, the director of equity and race relations for Seattle Public Schools.

“We know that cultural competence is an important aspect of being an effective teacher, but how do you measure that?”

“Thus, no research has shown that teacher’s cultural competence is associated with better student learning.”

five topics
Five topics
  • measurement theory
  • reliability
  • validity
  • measurement selection
  • measurement construction
measurement theory
measurement theory
  • levels of measurement
    • N O I R
  • Consequences of level of measurement on analysis choices
measurement theory5
measurement theory
  • “fundamentalist” or representational view:Stevens:
    • measurement scales are direct representations of what they measure
    • a measure of an attribute can be classified into distinct levels (N,O,I or R)
    • analyses are only permitted as limited by the level of measurement
measurement theory6
measurement theory

a fundamentalist “map” of permissible analyses

measurement theory7
measurement theory
  • operational or statistical viewNunnally, Gaito, etc.:
    • a score does not represent the trait, it is just a measure of the trait
    • but each score has error and is thus only an estimate
    • permitted analyses depend on the set of items used to minimize error (reliability) as well as considerations for the underlying nature of the phenomenon being measured (intrinsically N, O, I R?
measurement theory8
measurement theory






unique measurement variance

item 5

item 4

item 3

item 2

item 1


common measurement variance

measurement theory9
measurement theory
  • Under operational theory…
  • Likert and other similar scales fundamentalists would consider ordinal can be analyzed using parametric procedures if
  • the set of items reflect a reliable assessment of the construct and the underlying concept under study is likely interval or ratio.
  • In doing this we assume that the numbers we assign the ranked categories approximately assess the true (but unknown) intervals between them.

Does the measure give consistent results?

  • inter-rater
  • test-retest
  • parallel-forms
  • internal consistency

Is the measure assessing what it claims?

  • face
  • content
  • criterion (predictive & concurrent)
  • construct
four typical modes of measurement in social research
Four typical modes of measurement in social research
  • archival assessment – example FirstSteps Database
  • self-report/self-administered – example from Ages and Stages Questionnaire
  • interviews – example from SSDP Interview
  • behavioral observation - example: TIP Observational Coding System
ssdp intergenerational project social development research group

SSDP Intergenerational ProjectSocial Development Research Group

Karl G. HillPrincipal Investigator

J. David HawkinsInvestigator

Richard F. CatalanoInvestigator

Robert McMahonInvestigator

Lewayne GilchristCollaborator

Jennifer BaileyResearch Analyst

Sabrina OesterleResearch Analyst

Maria Roper-CaldbeckData Collection Supervisor

Karen SegarData Manager

Amber TabaresGraduate Research Assistant

Erica BourgetProgram Coordinator

method measures
Method & Measures

TIP Sample

  • The SSDP sample parent,
  • His or her eldest biological child,
  • A second primary caregiver (most often the spouse) identified by the sample parent as the person who shares the greatest responsibility for raising the child.


  • Year-round data collection where the child and caregivers are interviewed +/- six weeks around the child’s birthday.
  • In-person interview and observation session consisting of:
    • parent interview
    • parent child-interaction task (age 2+)
    • other caregiver interview
    • other caregiver-child interaction task (age 2+)
    • child interview (age 6+)
tip descriptives
TIP descriptives
  • G3 Sample Size
    • 258 Families/children
    • 173 SSDP mothers, 85 SSDP fathers
    • 132 girls and 107 boys
    • 40% age 1-5
    • 60% age 6-13

95% retention in Waves 2 and 3

  • G3 Ethnicity
    • 33% Caucasian American
    • 21% African American
    • 37% Multiethnic
    • 9% Asian, Hispanic, Native American
  • Family Structure
    • 70% two parents (95% of whom live together),
    • 20% were being raised by one parent alone,
    • 10% one parent and a non-parental caregiver.
selecting pre existing measures
Selecting pre-existing measures

Research the literature to determine how your constructs have been measured before.

Try to use established measures when possible.




reliability & validity info

Obtain copies for pretesting

Alter the assessments only if absolutely necessary.

Identify and define the constructs you want to measure

selecting pre existing data archival assessment example firststeps database
Selecting Pre-existing Data:Archival Assessment– example FirstSteps Database

The database links measures of pregnancy outcomes to descriptions of maternity care services (from Medicaid payment records for Medicaid clients in Washington State) and background information on the mother's health and socio-demographic status (from birth and death certificates for all Washington residents).

archival assessment other kinds of archival records
Archival Assessment Other kinds of archival records


  • school records (grades, achievement test, disciplinary action, etc.)
  • police, court & detention records
  • health?


  • schools, neighborhoods, etc.
selecting pre existing measures21
Selecting pre-existing measures

These guidelines should be kept in mind in selecting pre-existing measures:

  • What level of measurement does it provide?
  • Possible Bias?
  • Is it reliable?
  • Has it been validated?
self report self administered assessment ages and stages questionnaire tip23
Self-report/Self-administered AssessmentAges and Stages Questionnaire (TIP)
  • Item Selection
  • ASQ items were developed using a variety of sources, including standardized
  • developmental tests, nonstandardized tests focused on early development,
  • textbooks, and other literature containing information about early developmental
  • milestones. Using these sources, the following criteria were used to
  • develop items:
    • 1. Skills were selected that could be easily observed or elicited by parents.
    • 2. Skills were selected that were highly likely to occur in a home setting.
  • Once skills were selected, items were written in familiar, nonjargon
  • wording not to exceed a sixth-grade reading level, illustrations were provided
  • when possible, and concrete examples were provided as appropriate.
scale item measure development when there are no good pre existing measures of the construct
Scale/item/measure development:when there are no good pre-existing measures of the construct.

broad qualitative studies

focusgroups togenerateitems

final scaleimplementation

theory /research /existing measures

item and scale construction

pretesting &reliability &validity analyses

constructing measurement instruments schaeffer pressler 2003 r b ch 7
Constructing Measurement Instruments (Schaeffer & Pressler 2003; R&B Ch 7)
  • Guidelines for asking questionsquestions/statements; clarity; avoid double-barreled items; reading level; relevant; short; phrase in positive; avoid bias;
  • Questionnaire construction
  • Composite Measures
  • Scaling Procedures
example structured interview seattle social development project
Example Structured InterviewSeattle Social Development Project


Prosocial involvement

Prosocial rewards

Bonding to prosocial others

Prosocial Beliefs

External constraints:Norms

Family & Classroom Management

Positive Behavior

Skills for interaction

Position in the social structure: race, SES, age, gender

Problem Behavior

Individual constitutional factors

Antisocial opportunities

Antisocial involvement

Antisocial rewards

Bonding to antisocial others

Belief in antisocial values


( +, -

( - )

( + )

Social Development Model


Prosocial opportunities

interviews example from ssdp interview
Interviews – example from SSDP Interview

Work Opportunities

  • There are a lot of chances for people like me to get a job with a future.
  • There are a lot of chances to move ahead or advance in my job.
  • These days it's just about impossible for someone like me to find a job.
  • The only jobs available for someone like me are dead-end jobs.
  • I have a lot of chances to do things with my coworkers.
  • I have lots of chances to do new things in my job.
  • I have lots of chances to make a contribution in my job.

Work Involvement

  • How often are your involved in making significant decisions at work?
  • How often do you interact, or talk with your coworkers?
  • How often do you supervise others on your job?
  • How much responsibility do you have in your job? [Would you say…]
  • I put a lot of energy into my job.

Work Rewards

  • You feel that you are making a useful contribution to society.
  • I gain a sense of accomplishment from my job.
  • My supervisors recognize when I am doing a good job and let me know about it.
  • My job is rewarding.
scaling procedures
Scaling Procedures

SPSS example

  • Evaluation of items
  • Reliability Analysis
  • Validity Analysis
reliability and validity analyses see handouts
Reliability and Validity Analyses: see handouts


Construct alpha 1 2 3

1 Work Opportunities .77 -

2 Work Involvement .66 .45 -

3 Work Rewards .80 .57 .47 -

  • Semi-structured (interviewer has a general structure and common prompts to move the interview along to elicit themes of interest to the study). Used in more qualitative studies.
  • Structured(interviewer has a determined structure of questions to ask, prompts, etc.)
observational coding advantages of observational coding
Observational CodingAdvantages of Observational Coding
  • Theorywitness behavior unavailable to other methods
  • Analysismulti-modal assessment to reduce bias
  • Prevention/Interventionobservations are tied more closely to the identification of specific behaviors that could be targeted for intervention
social development model observational coding system
Social Development Model—Observational Coding System
  • Theory driven
  • Observed, in-home, parent-child interactions
  • For families with children age 2 to 18 yrs
  • Demonstrated predictive validity
the social development sdm observational coding system
The Social Development (SDM)Observational Coding System
  • Two coding systems have been developed
    • One to code families with children 2-8 years old.
    • Another to code families with children ages 9+.
  • Parallel SDM constructs are used in both systems.
  • Micro- and macro-level ratings are included in the SDM-OCS.
these macro coded scales reflect sdm and other constructs of interest
These macro-coded scales reflect SDM and other constructs of interest

External Constraints

  • prosocial guidelines
  • antisocial guidelines
  • coercive parenting


  • prosocial opportunities


  • dyad’s involvement in the activity
  • parent’s prosocial involvement
  • parent’s antisocial involvement
  • child’s prosocial involvement
  • child’s antisocial involvement
  • dyad’s positive involvement


  • consequences for antisocial involvement
  • consequences for prosocial involvement



  • followed directions
  • parent’s problem-solving skills
  • child’s problem-solving skills
  • dyad’s problem-solving skills
  • parent’s recognition skills
  • parent’s verbal skills
  • child’s verbal skills
  • general parenting skill
training of coders
Training of Coders
  • Noldus Observer software is used to code videotaped family discussions
  • 160-175 hours are required to reach an acceptable level of interrater agreement
  • Training involves extensive discussion of cultural differences in behavior and in the meaning of behavior
  • All families in the examples have provided consent to use their observations for training and research purposes outside of the specific project.
tasks for children 2 8 years
Tasks for Children 2-8 years
  • Child’s Game
  • Caregiver’s Game
  • Clean Up
macro code child 2 8
Macro-Code: Child 2-8

Child’s Game (5 minutes): Parent is asked to play with child, following the child’s lead.

  • child antisocial involvement
  • low general parenting skill
macro code child 2 843
Macro-Code: Child 2-8
  • opportunities for prosocial involvement
  • child prosocial involvement
  • rewards for prosocial involvement

Caregiver’s Game(5 minutes): Parent is asked to have the child follow what they want to play with.

tasks for children 9 years
Tasks for Children 9+ years
  • Expectations and Guidelines -10 minutes
  • Solving Family Problems – 10 minutes
  • Recognition- 5 minutes
external constraints expectations and guidelines
External ConstraintsExpectations and Guidelines
  • Parent and Teen are Asked to Discuss rules/guidelines
  • Guidelines
    • positive
      • During the task, caregiver communicated clear expectations/rules/ guidelines about the child/teen’s use of (tobacco, alcohol, drugs, sexual activity, violence)
skills for involvement recogniton
Skills for InvolvementRecogniton
  • Positive
    • Father was able to identify a positive attribute about the child/teen.
    • Family members demonstrated genuine warmth and caring toward each other.
reliability and predictive validity ssdp intergenerational project
Reliability and Predictive Validity: SSDP Intergenerational Project

Prosocial Guidelines 0.84 -0.47**

Prosocial Opportunities 0.82 -0.29+

Dyad Involvement In The Activity 0.78 -0.30*

Child Prosocial Involvement 0.86 -0.24

Dyad Positive Involvement 0.92 -0.37*

Consequences For Prosoc. Invol. 0.82 -0.27+

Bonding 0.82 -0.22

Followed Directions 0.64 -0.35*

Parent Problem-solving Skills 0.83 -0.35*

Child Problem-solving Skills 0.74 -0.32*

Dyad Problem-solving Skills 0.85 -0.44**

Low Skilled Parenting 0.74 0.51***

Corr. with Parent

Macro-Code Scale alpha Binge Drinking