IHSN Question Bank - supporting quality and consistency in survey data

1. IHSN Question Bank- supporting quality and consistency in survey data Olivier Dupriez World Bank François Fonteneau PARIS21/OECD Geoffrey Greenwell PARIS21/OECD Mark McConaghy DFID - UK International Household Survey Network

2. Evidence informed policy “Trusted, high quality statistics are a necessary condition for developing and maintaining a strong economy and society that functions well” (Hon. Prof. Peter Anyang’ Nyong’o former Minister for Planning and Development, Kenya) “Comparing groups, cultures, nations or continents is an essential means of distinguishing between local conditions and universal regularities” (Janet Harkness; Cross-Cultural Survey Methods; 2003:3)

3. Questions and concepts Disease prevention Concepts Latent constructs Improved/ unimproved drinking water sources Access to clean/safe water Indicators Questions What is the main source of drinking water for your household?

4. ….but different building blocks

5. Changes to questions

6. Deprivation estimates in same country ? Different illnesses included- untreated diarrhoeal illness verses ARI Different ways of treating household and communal information

7. Measuring educational attainment All aged 5 or over Has (name) ever attended school <If YES> What was the highest school level attended? Within that school level, what was the highest grade completed? Can you read and write? <If YES and age greater than 13> What was the highest school level you attended? Within that level, what was the highest grade completed?

8. Educational attainment outputs Biased up and down by those who cannot read & write but did attend some schooling?

9. Documentation for replication “The only way to understand and evaluate an empirical analysis fully is to know the exact process by which the data were collected and the analysis produced… The replication standard holds that sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party could replicate the results without any additional information from the author” (Gary King in Political Science and Politics Vol. XXVIII No 3 443-449)

10. Example DDI elements Literal question –What kind of toilet facility do members of your household usually use? Answer categories Flush/pour flush Flush to piped sewer system 11 Flush to sceptic tank 12 …. Ventilated Improved Pit latrine 21 Pit latrine with slab 22 Pit latrine without slab/open pit 23 …. Post question text If ‘flush’ or ‘pour flush’ probe – Where does it flush to? IF NECESSARY ASK PERMISSION TO OBSERVE THE FACILITY

11. DDI helps with interoperability Notes for Interviewer Pit latrines are the best conventional sanitation option when there is no source of water on or close to the plot so that water use is low (typically less than 25 liters per person per day). They may also be used when water consumption is higher, provided that separate provision is made for sullage (“gray” water) disposal. However, they may be less attractive to users of pour-flush toilets in such situations. Simple unimproved pit latrines may smell. VIPs should be better but users may not appreciate the need for a dark interior and may modify the superstructure to provide more light, thus undermining the basic rationale behind the design. The VIP design is very dependent on a durable fly screen and may give rise to insect problems if the screen fails for any reason. High water table and/or flooding are obvious problems for pit latrines. Universe All households Source of information Responsible household member

12. Inclusion of technical material

13. Ongoing and planned work, so far… UNICEF/WHO immunization, anthropometry, nutrition etc Joint Monitoring Program water & sanitation FAO/ World Food Program Agricultural census and surveys ILO economic activity, Int. labour migration World Bank/ ILO consumption, savings & assets, remittances World Bank/ ONS social capital UIS/UNESCO educational surveys IHFAN health facility surveys

14. Question Bank – key components • Using DDI to develop a meta data structure for data collection a priori, link concepts, indicators & questions • Central repository for survey guidelines - custodians for topic areas (labour, health, food, education …) ‘a living resource’ • Generation of interviewing, editing and coding instructions • Standards for coding and classification • Transparency with derived variables/ indicators creation • Tabulation plans for a primary analysis

15. Question Bank to include discussion forum Facilitate production of questionnaires and data input programs including CAI? Provide the application to NSOs to help establish Question Banks at country level Introduce different language versions, translation issues and cognitive testing/piloting Develop information about relationship between concepts and questions across countries and cultures Question Bank – possible extensions

16. Technical aspects of the Question Bank • XML schema has been developed to store all information (built on DDI and Dublin Core standards, plus additional elements) • Q-Bank will be a user-friendly application that makes re-use of metadata easy (built on Flex/Air). It will be web-enabled and also work as a stand-alone application. • user-friendly "metadata editor" developed to ensure easy maintenance of the system • all applications are open source, and will be distributed free of charge • XML schema and the application support multi-lingual metadata • application can serve as a repository for international or national classifications

17. Question Bank important because “Careful documentation of design decisions and implementations will enable cross-cultural researchers to begin accumulating a knowledge base of experience that will promote greater awareness of the relative advantages of each approach and move the community closer to establishing ‘best practice’ recommendations with confidence” (Janet Harkness; Cross-Cultural Survey Methods; 2003: 34)

18. Thank youwww.surveynetwork.org International Household Survey Network