Scanning Technology and Its Application in Ethiopia Yakob Mudesir Deputy Director General

1. Scanning Technology and Its Application in Ethiopia Yakob Mudesir Deputy Director General Central Statistical Agency of Ethiopia E-mail:yakobm@ethionet.et Web:www.csa.gov.et October 2009 Kampala, Uganda

2. Outline: • Background • Scanning Technology • Requirements for Scanning • The Ethiopian Experience

3. Background • Population and Housing Census process is the largest data capturing exercise that a country can undertake. • It involves capturing of millions of forms • The Central Statistics Agency (CSA) started using old techniques like Punched Card Reader as early 1960’s. • Two Population and Housing Censuses have so far been conducted in Ethiopia using the traditional method of data capturing. • The first Population and Housing Census was carried out in 1984.

4. Background • During the 1984 Census: • Data capture was done on manual keyboard based entry using mainframe computer • FORMSPEC data entry system was used • It took more than 2 years to capture the data for about 42 million people. • In the case of the 1994 Census: • Data capture was again done on manual keyboard entry basis using PC’s • CENTRY data entry system (IMPS) was used

5. Background • It took about 18 months to capture the data for the population of about 53 million. • The entry work was done on 2-shift basis • About 180 data entry clerkswere involved • Around 90 Pc’s were used

6. Some Limitations of the Keyboard Manual Entry Method • Time consuming • Does not allow the availability of timely data • The data will be weaker in representing the current or existing situation • Subject to additional non-sampling errors • Human error due to manual keying • Due to the volume of the data, a 100% verification as in the case of sample surveys, is difficult.

7. Some Limitations of the Keyboard Manual Entry Method • Involves a great deal of human resource management. • Large number of data entry operators and equipment required

8. The need for alternative solution The need to have the census result right on time and the limitations discussed above forced the statistical offices to look for other alternatives.. specially related to large volume of data - The Emerging of Scanning Technology

9. The Scanning Technology • The Scanning Technology in general implements two basic techniques • Mark recognition, like the Optical Mark Reader (OMR) • Character recognition, like the Optical Character Recognition (OCR) and the Intelligent Character Recognition (ICR)

10. The Scanning Technology • OMR is the recognition of shaded marks (blobs) on the forms • The positioning of these blobs on a form • determines the alphanumeric characters • they represent • The character recognition is the recognition of alphanumeric characters on forms and they are of 2 types: • OCR which is the recognition of machine printed characters and . . • ICR which refers to the capture of hand- • printed characters from a form

11. Major Benefits of the Scanning Technology • Significant decrease in time required to capture the data • This helps to get timely data • Users’ need satisfied (policy makers, planners, researchers, etc.) • Reduces the non sampling error • No need to worry to store millions of forms for possible future references • Scanning captures the whole content of a questionnaire in an electronic image format

12. The Process involved in Scanning Recognition & Extraction Scanning 1 1 33 35 33 EXPORT 1 2 1 2 07 1 01 2

13. Requirements for Effective Scanning - Proper planning - On the DESIGN of the questionnaire - On PRINTING the questionnaire - On RECORDING answer - On Questionnaire HANDLING - On securing ADEQUATE SPACE for questionnaire movement Why? In order to minimize the rejection rate and increase the Recognition Rate

14. Requirements for Effective Scanning • Proper training • Both on Hardware and Software • This helps to “ own” the technology • Being able to use the technology after the departure of the trainers / technical advisors • A well organized space for forms and data flow is required

15. Requirements for Effective Scanning - A Reliable Network System

16. Data Processing Center Receiving Warehouse Registering the EA for Scanning 1 Registering the EA Received from the Field 4 Waiting Room 3 5 Scanning Room 2 Receiving the Questionnaires 6 Store Validation Room 7 8 Processing Room Requirements for Effective Scanning - A WELL STRUCTURED SPACE FOR FILE FLOW

17. Requirements for Effective Scanning Proper File Management BOX File recorder: Records the ID of the outgoing questionnaires Filing Box handler Checks the village code on the box is the same as on the questionnaire Filing Box handler Returns the questioanires to the shelves File recorder: Records the ID No of the incomming questionnaires. Machine room box handler Checks the number, orientation and damage to questionnaires Principal Supervisor Machine room box handler returns box to the File recorder Paper handling cycle Senior Supervisor Guillotine Machine operator Cuts the binding edge off all questionnaires in the village in one operation and places them on the scanner Stitching machine operator lifts paper off scanner and stitches in the bottom left corner LoosePaper Scanner Paper Handler Responsible for checking paper throughput through the scanner Loose Paper Scanner Supervisor Scanner technician Ensures smooth running of the machine and assists with the paper handling Computer operators (x2) Responsible detecting errors in the scanning process

18. Requirements for Effective Scanning • Proper file management and care • Checking batch (EA) IDs and orientation of forms • Proper recording of the in-coming and out- going questionnaires • Ensuring the EA code on each box is the same as the one on the questionnaires • Close attention in detecting errors in the scanning process is required

19. The Ethiopian Experience

20. Experience Sharing • Study tour made in two African countries • Tanzania • To learn from their successes • Data capture of the 2002 Census of Tanzania was done in about 26 days • General report tables were produced within 3 months from the start of the scanning

21. Experience Sharing • Ghana • To learn from their difficulties • Data capture of the 2000 Census took about 6 months - ( forms from 29,000 EAs) • 3 Scanners were used (Kodak, Fujitsu) > The larger scanner was Kodak 500D > Speed: About 500 forms/min • Power failure was one of the major problems > Loss of some data occurred as a result > A large generator was installed to minimize the effect of the frequent power cut

22. Identification of the Technology • For scanning of the 2007 Census the Optical Mark Reader (OMR) technique was used • Scanning Technology to be used PhotoScribe Series PS900 Scanners DRS Scanning Technology product

23. Identification of the Technology • DRS Photo Scribe Series PS900 • High speed Imaging Mark Reader • Windows XP professional • CD R/WR drive • Network connectivity • A TFT monitor, Keyboard, mouse • Speed: up to 8,500 forms / hour

24. Design and Printing of Forms • Types of the 2007 Census forms • Short questionnaires • Long questionnaires • Household Listing Forms • Summary Forms • Community Level Forms • Batch Header Form – Scanned to create EA database

25. Design and Printing of Forms Long Questionnaire

26. Design and Printing of Forms Control Database Form Summary Form

27. Pilot • Data from the Pilot Census successfully scanned (OMR), key-corrected, exported to text format, tabulated and tested. • One scanner (PS 900 Photo Scribe) was used to capture the pilot data • Technical experts from the DRS company assisted in capturing, validating and exporting the pilot data

28. Training • Hardware and Software training • 16 professionals trained • The training in general took about 7 working days • SOSKITW for Windows:- a DRS software package for scanning was used • Components of the SOSKITW Software : • SOSGen : - used to generate scanning decodes for completed OMR forms (How marks on forms are interpreted and stored) • SOSInp : - used to scan, validate and export scanned data.

29. Preparatory Activities • Equipment purchased and installed • 10 additional PS900 iM2 DRS Scanners • 16 high capacity PC’s for key-correction • Census data processing work plan prepared • Recruitment of temporary staff • Staff training (scanning technology, CSPro) • Retrieval and organization of completed forms • Scanning and validation • Computer editing and tabulation (For each activity: duration and responsible body are indicated)

30. Preparatory Activities • About 33 teams for registering and organizing forms are organized • 3 persons assigned per team • Census data processing teams organized • Batch header database group • Scanning and validation team • Shift supervisors • Two senior programmers responsible for the overall scanning process • Other sub-professional staff assigned • 4 batch header scanning technicians • 16 data validation workers

31. The scanning room organized An air conditioner for the scanning room installed A high capacity automatic generator installed to ensure uninterrupted power supply Preparatory Activities

32. The Scanning Process • Organized forms taken from store to the waiting room • Batch header information printed and associated with its respective EA box • The existence of each EA verified • Checked EAs sent to the scanning room • Scanned forms are finally sent back to the stores • Scanned data are validated / key-corrected

33. The Scanning Process The actual scanning started mid July and the scanning work has been completed in November 2007

34. Data Cleaning / Computer Editing • Scanned, key-corrected and exported data • Batch Edit program based on edit specs provided by subject matter specialists developed and run on the data. • The software to be used in editing the data will be the Census and Survey Processing System (CSPro)

35. Future Plan • Owning the Technology • Two Professionals have been trained in England for two weeks on scanned document processing using DocXP in close collaboration with DRS • Printing of the Questionnaire locally • Questionnaire design • Our Professionals are working hard to process the upcoming welfare Monitoring Survey using scanning technology

36. THANK YOU