1 / 39

ABS Statistical Databases

ABS Statistical Databases. Session 6 Mark Viney Australian Bureau of Statistics. 6 June 2007. "Stove Pipe" approach. INPUT. THRUPUT. OUTPUT. INPUT. THRUPUT. OUTPUT. INPUT. THRUPUT. OUTPUT. "Clearing-House" Approach. THRUPUT. INPUT. OUTPUT. OUTPUT. ABSIW. INPUT. IDW. OUTPUT.

hester
Download Presentation

ABS Statistical Databases

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. ABS Statistical Databases Session 6 Mark Viney Australian Bureau of Statistics 6 June 2007

  2. "Stove Pipe" approach INPUT THRUPUT OUTPUT INPUT THRUPUT OUTPUT INPUT THRUPUT OUTPUT

  3. "Clearing-House" Approach THRUPUT INPUT OUTPUT OUTPUT ABSIW INPUT IDW OUTPUT OUTPUT INPUT OUTPUT Standardised Standardised Standardised interface interface interface

  4. e-Census

  5. e-Census 2006 • Conducted 2006 Population Census with the option of electronic submission of responses • drop-off/ pick up • drop-off/mail back in 2011 • 10.2% of returns were electronic • no edits incorporated into electronic form • less visits to pick up paper forms • less paper forms • less scanning/repair

  6. ABS Secure Deposit Box

  7. Secure Deposit Box • An externally facing database to allow respondents to lodge their raw data electronically • Excel spreadsheet (essentailly replacing a paper form) • Administrative datasets

  8. ABS Statistical Databases • ABS Input Data Warehouse (ABS IDW) • ABS Information Warehouse (ABSIW)

  9. ABS Input Data Warehouse (ABS IDW)

  10. Input Data Warehouse • Used as a repository for data as soon as it is entered into ABS computer systems • Initially used for data received electronically • Now used to load (and process) survey data

  11. Input Data Warehouse • Structure • Star schema • 1 fact table and several dimension tables • each data cell is stored as 1 row in the fact table

  12. Star Schema

  13. ABS Input Data Warehouse - What it allows us to do • Keep a historical record of what each cell was at every point in the processing • Reason for the change • when it changed • who changed it • change in value • Ready access to both current and historical data

  14. ABS Input Data Warehouse - What it allows us to do • A data store for use with :- • editing • imputation • winsorisation • estimation • Quick easy analysis and confrontation of data:- • across time • across dataitems • across data sources

  15. ABS Input Data Warehouse - Flow of Information

  16. What we hope to achieve from IDW • Reduced costs • Improved data quality • Tools to assist with management of data providers • Better understanding of Editing processes • Significance Editing • One single source of microdata • for all statistical collections • Well managed and secure data storage

  17. ABS Information Warehouse (ABSIW)

  18. ABS Information Warehouse • Need to make both data and metadata:- • Visible • Relatable • Accessible • Understandable • Reliable • Media Independent

  19. ABS Information Warehouse • Visible • central known location • Relatable • across collections • Accessible • tools to allow extraction and manipulation

  20. ABS Information Warehouse • Understandable • data fully described by metadata • Reliable • single source • high availability • Media Independent • single source for outputs • paper publications • electronic releases • ad - hoc requests

  21. ABS Information Warehouse • Define and manage metadata • Load lightly aggregated data • Validate data as compliant with metadata • Manipulate data • Produce statistical outputs • Make data publicly available

  22. ABS Information Warehouse - Flow of information Load info about collection Disseminate time series Load data to the ABSDB Closed DB Sign-off data to the ABSDB Open DB Data from a collection Load info on how to categorize data Disseminate output tables Derive ad-hoc client data requests Load info on what data items mean Processing System Information Warehouse PPW

  23. ABS Information Warehouse - Define and Manage Metadata • Interfaces to manage metadata • load, amend, validate, extract • dataitems,classifications, collections,datasets,publications • Application Program Interfaces (API) to link with other systems/programs • increasingly using XML

  24. ABS Information Warehouse - Loading data • Load data from major sources • Input Data Warehouse • SAS • FAME • SuperCROSS

  25. ABS Information Warehouse - Generating New data Cubes • Passing data through one or more steps to derive a new table • aggregation • drop dataitems • calculate new items

  26. ABS Information Warehouse - Other Manipulations • Seasonal Adjustment • SeasABS (X-11) • Chain Volume Measures • FAME (timeseries) • Supertables • Confidentialisation • Disclosure Avoidance Analysis System

  27. ABS Information Warehouse - Data Delivery • Data combined with metadata • Output formats created tailored to specific use • spreadsheets • timeseries • supertables • paper publications • electronic release

  28. ABS Information Warehouse - Public Release • Make data available on an internally accessible database at a predetermined time (usually 11:30 am Canberra time) • This data is then available to ABS Statistical Consultants to satisfy customer requests • Feed data to website • www.abs.gov.au

  29. ABS Website www.abs.gov.au

  30. National Data Network (NDN) www.nationaldatanetwork.org

  31. Australian Bureau of Statistics We assist and encourage informed decision making, research and discussion within governments and the community, by providingleading a high quality, objective and responsive national statistical service

  32. National Data Network • Website that raises visibility of statistical data • regardless of publishing agency A national platform for acquiring, sharing and integrating data relevant to policy and research in Australia

  33. National Data Network • One central website • descriptions of data • quality statement • references to other data • Several websites (Nodes) owned and maintained by other agencies www.nationaldatanetwork.org

  34. National Data Network

  35. National Data Network • Current Focus • Publish / Search / Acquire • Future Focus • Design / Capture / Process • Analyse / Report

  36. Australian Bureau of Statistics We assist and encourage informed decision making, research and discussion within governments and the community, by providingleading a high quality, objective and responsive national statistical service

  37. Questions?

More Related