Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service

1. Identifying Sources of Error: the 2007 Classification Error Survey for the US Census of Agriculture Jaki McCarthy and Denise Abreu USDA’s National Agricultural Statistics Service Presented at the International Total Survey Error Workshop Tallberg, Sweden June 2009

2. Target: Census of Agriculture Alternate Source of Information: June Agricultural Survey Errors in one survey can be measured with matching information from other sources

3. Error of Interest: Scoping Errors, i.e. Census Misclassification • Census farms incorrectly classified as non-farms • Census non-farms incorrectly classified as farms

4. Census of Agriculture Census of Agriculture conducted every 5 years Count of all US Ag operations ($1000 or more in sales) Primarily mail data collection Data collected December - March June Agricultural Survey (JAS) Annual area frame based sample survey in June JAS is primarily face to face interviews Data collected in first 2 weeks of June JAS has been used to measure undercoverage and misclassification on census Errors in one survey can be measured with matching information from other sources

5. JAS – Area Frame Based

8. NASS Area Frame - SEGMENT • Theoretically complete sampling frame • No overlap or gaps • Segments of land sampled

9. NASS Area Frame – Segment Enumeration • Sampled segments divided into tracts representing unique land operating arrangements • In-person interviewers screen for whether a tract is part of an agricultural operation and, if so, collect crop and livestock information A B C H D G E F

10. Background:Previous Classification Error Studies Measured census classification error – records incorrectly classified as farms or non-farms and duplication Census records matched to JAS JAS was assumed as truth; differences between the two sources were designated as census misclassification Overall census misclassification error was estimated

11. Background:Previous Classification Error Studies Net classification error was small and was not used to adjust census numbers For these reasons, shift in study’s primary objective 11

12. Current Classification Error Survey To identify REASONS for discrepancies between the JAS and the Census Qualitative examination of why errors occur Classification errors Reporting errors also examined To provide information to improve quality of the data, reduce analyst review and editing

13. 2007 CES Objective Determine whether acreage/scoping differences are legitimate changes or errors Determine why people report incorrectly Determine if the forms were correctly processed

14. Methods Census records matched to JAS records Respondents records with scoping or acreage discrepancies were identified Respondents re-interviewed and asked to resolve discrepancies

15. Identifying Groups with Discrepancies

16. Discrepancies between Census and JAS Scoping differences: 18.4% of matched records had discrepancies in classification (~3% net classification error) Acreage differences: 37.2% of matched records had acreage differing by more than 25%

17. Methods 67 respondents were re-interviewed by enumerators in July 2008 Respondents reviewed questionnaires from both the 2007 Census and the 2007 JAS Then asked to identify which was correct and why they were different

18. Scoping Differences Which Source is Correct?

19. Scoping Differences Which Source is Correct? TRUE Census Misclassification

20. Scoping Differences – Census is Correct Number of Responses (n=39)

21. Scoping Differences – JAS is Correct Number of Responses (n=10)

22. Scoping Differences – Both Sources Correct True Change – reported correctly True Change – reported incorrectly Number of Responses (n=9)

23. Scoping Differences – Neither Source Correct Number of Responses (n=9)

24. Scoping Differences – Overall Summary by Category True Change – Correct True Change - Incorrect Number of Responses (n=67)

25. Summary – Scoping Differences Very few of these cases are real changes between JAS and the Census Census was correct more often than June Most discrepancies are actual errors June tracts screened out incorrectly Proxy respondents reporting incorrectly in JAS Specific types of land excluded (government program land, woods, rented)

26. Source Used to Report Acres * Multiple answers allowed

27. What did we learn about Census misclassification? • Classification error remains minimal and is probably smaller than previous estimates • JAS cannot be used as “truth” • Re-interview with resolution shows both the Census and the JAS have errors • JAS is not the GOLD STANDARD --personal interviews not always best way to get accurate responses • Some errors due to respondents and won’t be eliminated

28. Our external source had more errors than our target: • Recommendations to improve the JAS: • Avoid proxy respondents in JAS • Review of screening in JAS • Intensive re-screening of all non-ag tracts is in progress • Estimation of farms missing • from JAS • Capture/Re-capture • estimates in progress

29. To examine errors, you need a good measure