slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Clyde Tucker and Brian Meekins U.S. Bureau of Labor Statistics and Paul Biemer PowerPoint Presentation
Download Presentation
Clyde Tucker and Brian Meekins U.S. Bureau of Labor Statistics and Paul Biemer

Loading in 2 Seconds...

play fullscreen
1 / 21

Clyde Tucker and Brian Meekins U.S. Bureau of Labor Statistics and Paul Biemer - PowerPoint PPT Presentation


  • 102 Views
  • Uploaded on

Using Substantive Diagnostics to Evaluate the Validity of Micro-level Latent Class Indicators of Measurement Error. Clyde Tucker and Brian Meekins U.S. Bureau of Labor Statistics and Paul Biemer Research Triangle Institute. Background.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Clyde Tucker and Brian Meekins U.S. Bureau of Labor Statistics and Paul Biemer' - lionel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Using Substantive Diagnostics to Evaluate the Validity of Micro-level Latent Class Indicators of Measurement Error

Clyde Tucker and Brian Meekins

U.S. Bureau of Labor Statistics

and Paul Biemer

Research Triangle Institute

background
Background
  • Developed by Lazarsfeld (1950)—unobserved or “latent” variable drawn from relationships between two or more “manifest” variables
  • Lazarsfeld and Henry (1968) and Goodman (1974) extended mathematics of theory
  • Software for latent class analysis (LCA) developed (MLLSA, lEM, M-PLUS)
  • LCA used to study measurement or response error (VandePol and deLeeuw 1986; Tucker 1992; Van de Pol and Langeheine 1997; Bassi et al. 2000; Biemer and Bushery 2000; Tucker, et al. 2002, 2003, 2004, 2005, 2006, and 2008)
creation of manifest variables
Creation of Manifest Variables
  • Try to create at least three
  • Try to avoid direct relationships with outcome variable (expenditures, in this case)
  • Use LCA to “triangulate” them to produce a latent variable with more information than any one of them alone
statistical logic
Statistical Logic
  • In mathematical terms, when manifest variables A and B are not independent, the following relationship will not hold:
  • where i indexes the classes of A, j indexes the classes of B, πijAB is the probability an individual is in cell ij, πiA is the probability an individual is in class i, and πjBis the probability an individual is in class j.
statistical logic1
Statistical Logic
  • For the above expression to be true, A and B must be independent. The purpose of the latent variable X is to achieve that independence. Thus, the following latent class model is desired:
  • where t indexes the classes of X, πijtABXis the probability of being in cell ijt of the unobserved ABX table, πtXis the probability that an individual is in one of the mutually exclusive and exhaustive classes of X, πitAXand πjtBX are the conditional probabilities that an individual is in a particular class of A and B, respectively, given that a person is in a certain class of X. Equation (2) indicates that, within a class of X, A and B are independent.
purpose of paper
Purpose of Paper
  • Concept of LCA relatively straightforward—create a variable to account for common variance among observed variables
  • Issues:
    • What is the new variable?
    • What do its classes mean?
    • Does it really tell us anything useful?
  • Statistical diagnostics don’t help us here. We need substantive ones.
  • Paper explores some of this type of diagnostics
data sources
Data Sources
  • CED
    • 2 week diaries
    • All expenditures
    • Small items and grocery expenditures
    • Used for CPI cost weights
  • CEQ
    • 5 quarters (first for bounding) PV
    • All consumer expenditures
    • 2 hours
    • Larger consumer items
    • Used for CPI cost weights
three examples
Three Examples
  • 1985 CED Operational Test (micro level)
    • 3 treatments—specific, nonspecific, control
    • 800 households in each
    • Latent response error measure of underreporting of grocery expenditures using manifest performance indicators
  • CEQ (1996-2001) (micro level)
    • Only analyzed the 2nd wave
    • 43,000 completed 2nd wave interviews
    • Latent response error measure of underreporting for 7 expenditure categories for purchasers using manifest performance indicators
  • CEQ (1996-2001) (micro level)
    • Analyzed all four waves
    • 14,877 remained in sample throughout
    • Latent response error measure of underreporting for almost 30 expenditure categories for all households (purchasers and nonpurchasers) using manifest performance indicators and indicators of pattern of wave nonresponse
critical assumption
Critical Assumption
  • Response errors in CE only come from underreporting of expenditures and not overreporting
    • Tedious
    • Time-consuming
    • Recall problems
    • Lack of knowledge
methodological issues
Methodological Issues
  • Weighted vs. unweighted
  • Variances for complex sample design vs. SRS
  • Local vs. global maxima
  • Sparse cells (too many manifest variables)
  • Restricted vs. unrestricted models
  • Boundary problems (no overreporting)
1985 diary test
1985 Diary Test
  • Manifest variables
    • Difference in first and second week grocery expenditures
    • Difference in usual and average weekly grocery expenditures
    • Amount of expenditure information collected by recall
    • Respondent’s attitudes and behavior with respect to diarykeeping
  • Latent variable
    • 3 classes (low, moderate, high response error)
ceq micro level manifest indicators for first study
CEQ Micro-level Manifest Indicators for First Study
  • Interview level indicators considered:
    • Number of contacts
    • Ratio of respondents/household members
    • Missing income data
    • Type and frequency of records used
    • Length of interview
    • Ratio of expenditures in last month to quarter
    • Combination of type of record and interview length
indicator coding
Indicator Coding
  • #contacts (1=0-2; 2=3-5; 3=6+)
  • Resp/hh size (1= <.5; 2= .5+)
  • Income missing (1=present; 2=missing)
  • Records use (1=never; 2=single type or sometimes; 3=multiple types and always)
  • Interview length (1= <45; 2=45-90; 3= 90+)
  • Month3 expn/all (1= <.25; 2= .25-.5; 3= +.5)
  • Combined records and length (1= poor; 2= fair; 3=good)
latent variables
Latent Variables
  • Three-class latent variables (poor, fair, good reporting) for
    • Kid’s Clothing
    • Women’s Clothing
    • Men’s Clothing
    • Furniture
    • Electricity
    • Minor Vehicle Repairs
    • Kitchen Accessories
second ceq micro level study
Second CEQ Micro-level Study
  • Based on results of first CEQ study, analysis of purchasers and nonpurchasers together
  • Used Interviews 2-5 data.
  • Not limited to within-interview indicators
  • Developed model using all Interview 2 respondents
  • Latent variable is still intended to represent quality of reporting
new manifest indicators
New Manifest Indicators
  • Overall Panel level indicators considered
    • Number of completed interviews (1-4)
    • Attrition combined with # of complete interviews
    • Average number of commodity categories for which CU had expenditure
    • Number of interviews the ratio of third month expenditure to quarter was between .25 - .5
    • Panel averages of interview level indicators from first CEQ study
model selection
Model Selection
  • Ran both ordered (fixed or restricted ordinal constraints) latent class models and unordered.
    • Order was determined based on theoretical relationship between values of indicators and level of underreporting.
  • Ran all combinations of indicators in groups of 3 & 4, using 3 or 4 category LC variable for each commodity category & overall
  • Multiple iterations to avoid local maxima
  • Best model candidates were selected based on fit
  • From those candidates, models selected based on relationship of indicators to latent construct
application of model
Application of Model
  • For the final models for each commodity:
    • Each combination of indicators was assigned to a latent class based on probability of being in that class given the value of the indicators
    • Ran demographic analysis to identify characteristics of members of each latent class
    • Expenditure means were found for each latent class
    • Examined the pattern of mean expenditure and the contribution of the latent variable in predicting these expenditures
second ceq micro level study expenditure categories
Cable/satellite TV

Men’s apparel

Women’s apparel

Men’s clothing only

Women’s clothing only

Men’s accessories

Women’s accessories

Men’s shoes

Women’s shoes

Kid’s apparel

Kid’s clothing only

Kid’s Accessories

Kid’s shoes

Dental care

Drugs and medical supplies

Electricity

Gas (household)

Eye care

Sports equipment

Televisions, video, & sound equip.

Vehicle service, major

Vehicle service, minor

Vehicle service, oil changes only

Vehicle expenses, other

Pets and pet supplies

Sports equipment

Trash collection

Televisions, video, & sound equip.

Vehicle service, major

Vehicle service, minor

Vehicle service, oil changes only

Vehicle expenses, other

Pets and pet supplies

Kitchen accessories

Other household items

Second CEQ Micro-level Study– Expenditure Categories
conclusions
Conclusions
  • When doing LCA for measuring response error, one cannot rely on statistical diagnostics alone. Substantive diagnostics are needed to judge the meaningfulness of the results.
  • Sometimes the models work and sometimes they don’t. Unfortunately, this is likely to depend on the characteristic you’re analyzing.
  • We need better manifest variables to explain more variance.
  • We have been unable to develop meaningful latent variables with more than three or four categories, and, in some cases, we could only identify two. LCA software really does work best with large sample sizes.
  • Besides only defining a few latent classes, we certainly will not progress beyond the most rudimentary ordinal rankings any time soon.
  • LCA problems are likely to be multiplied many times for response error measures for non-factual items such as attitudes or opinions.
clyde tucker senior survey methodologist osmr 202 691 7371 tucker clyde@bls gov www bls gov

Clyde TuckerSenior Survey MethodologistOSMR202-691-7371tucker.clyde@bls.govwww.bls.gov