Collecting online: How Statistics Netherlands moved its business Surveys onto the internet

1. Collecting online: How Statistics Netherlands moved its business Surveys onto the internet Ger Snijkersthanks to Deirdre Giesen, Hank Hermans, Myra Wieling

2. Overview Overview of developments at Statistics Netherlands:a. Electronic data collectionb. Organising data collection and data sharing Case study: Moving the Structural Business Survey to the weba. Developing and testing the questionnaireb. Communication strategy to launch the survey and improving web-take up ratesc. Implementation of the survey Lessons learned and challenges Looking into the future: future challenges OSS Seminar, Statistics New Zealand, Wellington

3. Brief history of electronic data collection at SN Before 2000: 1984 First PCs Mid 1980s Blaise 1.0 1993 E-reporting for International Trade: IRIS- disk-by-mail (floppy disks, CDs), - now offline using downloadable software Mid 1990s Electronic Data Interchange: EDIsent - software installed on business’ systems 1998 Simple establishment surveys - e-mail OSS Seminar, Statistics New Zealand, Wellington

4. E-mail questionnaire Business Tendency Survey 1999 OSS Seminar, Statistics New Zealand, Wellington

5. Brief history of electronic data collection at SN (cont) After 2000: 2000 Short Term Statistics (STS)- first electronic forms (HTML) 2003 Transportation Survey 2006 STS – New HTML forms 2006 Structural Business Surveys- Pilot offline 2007 Structural Business Survey in the field - offline downloadable software 2009 Business Surveys online: Transportation Survey 2014 - New integrated system for online surveys - Blaise 5 Next EDI using XBRL integrated in questionnaires 2005 Experiments for household surveys (on-line) 2008 Household surveys on the internet (on-line) OSS Seminar, Statistics New Zealand, Wellington

6. Transportation Survey 2003 OSS Seminar, Statistics New Zealand, Wellington

7. Offline SBS Survey 2007 more to come OSS Seminar, Statistics New Zealand, Wellington

8. Online Transportation Survey 2009 OSS Seminar, Statistics New Zealand, Wellington

9. XBRL-based system (future) OSS Seminar, Statistics New Zealand, Wellington

10. Health Survey 2010 OSS Seminar, Statistics New Zealand, Wellington

11. Business Data Collection at the moment A few numbers:  450 different questionnaires  400 mixed-mode: electronic & paper > 500,000 business units every year > 750,000 questionnaires on average each year  700,000 electronic questionnaires available OSS Seminar, Statistics New Zealand, Wellington

12. Drivers for using Web for Business Surveys From the data collection agency perspective:- data quality: cleaner (fewer edits) and more complete- cost-efficiency: cheaper and less editing- timeliness: faster From the business perspective:- internet penetration for businesses is high- much of the data held by businesses is already in digital form- many of the regulatory and administrative tasks they perform are already online- can be easier to do than paper (if designed well)  This only holds is surveys are designed well!- the whole communication process: from end to end OSS Seminar, Statistics New Zealand, Wellington

13. Conclusions so far Until 2007: Each survey for its own: stove-pipe approach No coordinated/harmonised design Have it in the field quickly Focus on technology, not on methodology/communication No efficient questionnaire design and communication: Internally: every survey starts all over, no learning curve Externally: no professional questionnaire and survey communication:- various designs - respondents have to learn every questionnaire again OSS Seminar, Statistics New Zealand, Wellington

14. Lessons learned Analyse contextual factors:- Internal constraints and processes- Response processes within businesses Focus on usability = success factor:- internally AND externally - technology AND methodology perspective “We suspect that many of the survey organisations that introduce web questionnaires forget that it is not the technology in itself, but how it is utilised that determines the result” ... both internally and within businesses From: Haraldsen & Couper (2013) Apply Deming cycle – PDCA:- also C & A: Evaluate and Improve OSS Seminar, Statistics New Zealand, Wellington

15. Lessons learned (cont’d) Good design is to create benefits for both sides: - Create internal conditions to be successful (under control) > One Data Collection Division - Tailor to business conditions (not under control; try to influence) > Moving SBS to the Web OSS Seminar, Statistics New Zealand, Wellington

16. Overview Overview of developments at Statistics Netherlands: a. Electronic data collection b. Organising data collection and data sharing Case study:Moving the Structural Business Survey to the weba. Developing and testing the questionnaireb. Communication strategy to launch the survey and improving web-take up ratesc. Implementation of the survey Lessons learned and challenges Looking into the future: future challenges OSS Seminar, Statistics New Zealand, Wellington

17. Data Collection over the years Stove pipes Data Collection 2000 Partial Centralisation Data Collection Now < 1994 1994 - 2000 2000 - 2007 Present OSS Seminar, Statistics New Zealand, Wellington

18. Organizing Data Collection in one DivisionProcess-and-knowledge Driven approach • Fte’s:- 375 (2008)- 277 (2012) • Budget:€ 18.5M • 145 surveys • Survey deployment: • CAPI / CATI Interviewing • Planning & control • Fieldwork logistics • Support • Special services • Front desk: • Client relations • Survey design 12 fte 240 fte 92 26 118 fte • Design: • Questionnaire • Sampling • Training 35 fte 122 fte Clustering of practices and knowledge • Focus on how knowledge is being used, for social and business surveys • Pilot 2007; effectuated 2011 OSS Seminar, Statistics New Zealand, Wellington

19. Drivers for centralizing data collection Internal drivers Achieving efficiency by: ∘ Abolishing redundant processes, workflows, and activities for social and business surveys- Monitoring processes, workflows, and activities ∘ Abolishing redundant systems and tools- One data collection management system - Maintaining as less systems and tools as needed Implementation of Data Collection Strategy (2005) External drivers: Professional external focus Reducing response burden OSS Seminar, Statistics New Zealand, Wellington

20. The 2011 Data Collection Strategy:Retrieving and returning (first version in 2005) 3 steps: 1. Re-use of available data- Data sharing & data warehousing 2. Use of new registers and other secondary sources- Traditional government-based registers - Big/organic data - Data on the internet (web-crawlers) 3. Primary data collection: 1. EDI technologies, like XBRL 2. Web surveys 3. traditional modes: paper, CATI, CAPI• Using new technologies• Reciprocity: report back to respondents Multi-source designs Mixed-mode designs OSS Seminar, Statistics New Zealand, Wellington

21. Data collection: Developments over the yearsSummary and conclusions From stove-pipes to a coordinated system of data collection From uni-mode surveys to multi-source/mixed-mode data collection designs From single-survey managers to managers of integrated sets of statistics From local decision making to corporate decision making From instrumental data collection to a communication perspective  We are still in the middle of this process! A cultural shift needs years to change! But … are we moving too slow? OSS Seminar, Statistics New Zealand, Wellington

22. Overview Overview of developments at Statistics Netherlands:a. Electronic data collectionb. Organising data collection and data sharing Case study: Moving the Structural Business Survey to the weba. Developing and testing the questionnaireb. Communication strategy to launch the survey and improving web-take up ratesc. Implementation of the survey Lessons learned and challenges Looking into the future: future challenges OSS Seminar, Statistics New Zealand, Wellington

23. Moving to the Web: two basic approaches Copy all paper questionnaires to the web in a short period of time, in one or various ways:- PDF, offline electronic questionnaire, online questionnaire  Quick results, but how is the questionnaire utilised?  High risks of: - web Qs not being used - negative effects on public image  Pushing problems and costs downstream  Evaluate and improve, if you still get the chance ... Develop one approach and apply to all surveys:- investing in the organisation, technology, and methodology- investing in the ‘respondent experience’: survey communication - investing in PR: harmonised approach  Takes time and costs money  Higher chances of good results: web Qs actually used OSS Seminar, Statistics New Zealand, Wellington

24. Moving the SBS questionnaire to the web: approach Approach by Stats Neth.: 2nd approach Develop one approach:- Good results for SBS …  No systematic evaluations and improvements that is applied to all surveys:Some follow-up, but not consistent, due to:- organisation not yet ready (one of the main factors for project failure)- still existing stove-pipes  Needs a long-term vision that is followed up  Assurance that investments are not lost Do it right: create benefits for both sides (or don’t do it all)! It is not going to be easy! OSS Seminar, Statistics New Zealand, Wellington

25. Moving the SBS questionnaire to the web: Project Project goals and constraints: • Develop a web questionnaire- same contents- mixed-mode design: coherent paper and web- support completion process- motivate respondents to use this mode- into the field: March 2006 • Start of project: June 2004- we had one year, until Sept. 2005 OSS Seminar, Statistics New Zealand, Wellington

26. Dutch Annual Structural Business Survey Some characteristics: Annual survey of economic activity, three parts:- revenues and costs - summary of business accounts: profits and losses - industry specific specifications Mandatory 75.000 business each year: -Sample of small businesses, bigger businesses each year Until 2006: paper questionnaire (A4 booklet):- Length may differ, 20 pages is typical- Items are grouped in sections- Completion process: complicated and difficult: lots of details asked for, mismatch of definitions, various business departments involved, design issues OSS Seminar, Statistics New Zealand, Wellington

27. Vernieuwde huisstijl:2. Onderzoek naar de PS Q oud OSS Seminar, Statistics New Zealand, Wellington

28. Vernieuwde huisstijl:2. Onderzoek naar de PS Q oud OSS Seminar, Statistics New Zealand, Wellington

29. 3. 4 projects3. Improving paper questionnaire Q nieuw OSS Seminar, Statistics New Zealand, Wellington

30. Developing the electronic questionnaire Developing and testing in five stages: 1. Develop and test the prototype (31-1-’05)- pre-tests to test usability: 3 waves - field agents, business respondents 2. Revision of questionnaire (1-9-’05)- expert reviews 3. Testing of revised questionnaire (1-1-’06)- additional usability tests 4. Implementation of field pilot (1-3-’06) 5. Implementation of survey (1-3-’07) OSS Seminar, Statistics New Zealand, Wellington

31. The electronic questionnaire: the prototype Developing paper Q and testing this prototype helped in thinking about a good design for an eQ OSS Seminar, Statistics New Zealand, Wellington

32. Testing the prototype Research issues: 1. How does the e-Q work in practice?- Completing the questionnaire- Question-and-answer process- Usability • What features should be included to make it easy to use?- Respondent friendly: ‘Computer-assisted’ tools- User demands 3. How should the eQ be designed in relation to the paper Q?- The same or a different design- ‘look-and-feel’ of paper and e-form OSS Seminar, Statistics New Zealand, Wellington

33. Testing the prototype: Research issues 1. How does the Q work? Laborious and complex process • Long, complex questionnaire (≥ 25 items) • Complex completion process:- several sessions, several informants- kick-and-rush behaviour • Imagine ... a respondent sitting behind his/her computer ...  Respondent got lost in the questionnaire OSS Seminar, Statistics New Zealand, Wellington

34. Illustration of usability “Medieval tech support” A new systemis introduced:The ‘book’ (“beek”), in stead ofrolls of paper. The monk Ansgard has called the helpdesk and is waiting for someone to come … OSS Seminar, Statistics New Zealand, Wellington

35. Illustration of usability “Medieval tech support” It illustrates what we found in the pre-tests for the electronic questionnaire … OSS Seminar, Statistics New Zealand, Wellington

36. Testing the prototype: Research issues 2. Features to make it easy to use? • What am I supposed to do (next)?- Easy to download, install, complete, send data back- It is one process: going to the internet – submitting data, - Clear instructions and explanations (but not read) • How is the questionnaire built up? - Show how the questionnaire is structured: overview- Help to find the way in the questionnaire- No hidden rules, no unexpected functionalities • Where am I? What did I do so far?- Provide overview of the completion process- Clear navigation, no scrolling- Printing function • Usability, navigation and overview: design of Tax forms • Choose a design people know: don’t be original! OSS Seminar, Statistics New Zealand, Wellington

37. Testing the prototype: Research issues 3. Design of paper and web Q? The computer is different than paper • The web Q reacts to the respondent • Reading from the screen is different • Navigating and getting an overview works differently • Kick-and-rush behaviour, even stronger than on paper • The use of computer-assisted functionalities; • the respondent expects the computer to react  Start with paper, then add functionalities OSS Seminar, Statistics New Zealand, Wellington

38. Testing the prototype: Conclusions of pre-test waves • Visual design Clear and logical: “Don’t make me think!” Simple, transparent, consistent No hidden and unexpected functionalities • Support the completion process, end to end Other mode, other features, other visual design Use computer functionalities: automatic calculations Use layout and usability principles that are known to R (country specific) • Tailor to kick-and-rush behaviour Split Q into small sections, small tasks Short and clear explanations OSS Seminar, Statistics New Zealand, Wellington

39. Lessons learned: Web Design Guidelines Web questionnaire design is communication design Questionnaire designer = Communication designer Questionnaire communication design OSS Seminar, Statistics New Zealand, Wellington

40. The revised questionnaire Based on: • Pre-test results • Expert reviews Iterative process with - Professional designer - Questionnaire designers - Methodologists • A user-friendly design was put first, not the IT tool • New prototypes designed in Power Point OSS Seminar, Statistics New Zealand, Wellington

41. OSS Seminar, Statistics New Zealand, Wellington

42. The revised questionnaire: additional pretests • 10 concurrent in-depth interviews • Usability and user friendliness had been improved  respondents enjoyed working with the questionnaire  they could handle the task: they recognised the design and usability  even though … the task had not changed OSS Seminar, Statistics New Zealand, Wellington

43. Field pilot Goals: • Implementation of web questionnaire- test usability and completion process in the field: downloading – completing – sending-in data- test communication strategy to get high web take-up rates • Test the whole process- logistics for paper and web flows • Response rates and web take-up rates? • Data quality- data editing, mode effects? OSS Seminar, Statistics New Zealand, Wellington

44. Field pilot Set-up: • March-July 2006 • 7200 businesses, 5 industries • Communication strategy: - internet portal: www.cbs.nl/productiestatistiek- advance letter with user name and password- leaflet to introduce web questionnaire and explain why the survey is conducted- paper questionnaire not mentioned- only 2e or 3e reminding letters include paper questionnaireIn the Netherlands businesses have to report electronically to the Tax Office. OSS Seminar, Statistics New Zealand, Wellington

45. Leaflet • Informing respondents about new questionnaire • Indicating benefits • Telling what will happen with the data Showing some relevant survey results OSS Seminar, Statistics New Zealand, Wellington

46. Field pilot Results: • Web questionnaire worked well • Web take-up rate: 80%, total response rate: comparable • No mode effects, but more research is needed • Internal processes worked well, but we missed checking the link to data editing.  SBS questionnaire and communication design was implemented by the end of 2006, and run in 2007 OSS Seminar, Statistics New Zealand, Wellington

47. Communication strategy: guidelines How to increase web take-up rate: Easy access to the web version combined with restricted access to the paper version Offer real advantages Making the respondents aware of the web option Offer practical guidance Computer competence and motivation play a minor part(if designed well) OSS Seminar, Statistics New Zealand, Wellington

48. For more guidelines … OSS Seminar, Statistics New Zealand, Wellington

49. Implementation of SBS Survey • Running from 2007 onward • About 75.000 business receive this questionnaire every year Web take-up rates: • Dutch Annual SBS, 2007: 84 %Total response rate did not change (80%) • Dutch average 2008: 51% 3 largest Dutch business surveys (2008): 77 %(STS: 63%, SBS: 80%, Intrastat: 100%) OSS Seminar, Statistics New Zealand, Wellington

50. https://formulieren.cbs.nl/downloads per survey, per period, per business unit the log on codes Implementation of SBS Survey OSS Seminar, Statistics New Zealand, Wellington