Data integration an overview on statistical methodologies and applications
Download
1 / 72

Data integration: an overview on statistical methodologies and applications. - PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on

Data integration: an overview on statistical methodologies and applications. Mauro Scanu Istat Central Unit on User Needs, Integration and Territorial Statistics [email protected] Summary. In what sense methods for integration are “statistical”?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Data integration: an overview on statistical methodologies and applications.' - hyatt-suarez


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Data integration an overview on statistical methodologies and applications

Data integration: an overview on statistical methodologies and applications.

Mauro Scanu

Istat

Central Unit on User Needs, Integration and Territorial Statistics

[email protected]

Poznan 20 October 2010


Summary
Summary and applications.

  • In what sense methods for integration are “statistical”?

  • Record linkage: definition, examples, methods, objectives and open problems

  • Statistical matching: definition, examples, methods, objectives and open problems

  • Micro integration processing: definition, examples, methods, objectives and open problems

  • Other statistical integration methods?

World Statistics Day


Methods for integration 1
Methods for integration 1 and applications.

Generally speaking, integration of two data sets is understood as a single unit integration: the objective is the detection of those records in the different data sets that belong to the same statistical unit. This action allows the reconstruction of a unique record of data that contains all the unit information collected in the different data sources on that unit.

On the contrary: let’s distinguish two different objectives - micro and macro

Micro: the objective is the “development” of a complete data set

Macro: the objective is the “development” of an aggregate (for example, a contingency table)

World Statistics Day


Methods for integration 2
Methods for integration 2 and applications.

Further, the methods of integration can be split in automatic and statistical methods

The automatic methods take into account a priori rules for the linkage of the data records

The statistical methods include a formal estimation or test procedure that should be applied on the available data: this estimation or test procedure

  • can be chosen according to optimality criteria,

  • and are associated with an estimate error.

    This talk restricts the attention on the (micro and macro) statistical methods of integration

World Statistics Day


Statistical methods
Statistical methods and applications.

Classical inference

  • There exists a data generating model

    2) The observed sample is an image of the data generating model

    3) We estimate the model from the observed sample

World Statistics Day


Statistical methods of integration
Statistical methods of integration and applications.

If a method of integration is used, it is necessary to include an intermediate phase.

The final data set is a blurred image of the data generating model

World Statistics Day


Statistical methods of integration1
Statistical methods of integration and applications.

Statistical methods for integration can be organized according to the available input

World Statistics Day


Record linkage
Record linkage and applications.

Input: two data sets on overlapping sets of units.

Problem: lack of a unique and correct record identifier

Alternative: sets of variables that (jointly) are able to identify units

Attention: variables can have “problems”!

Objective: the largest number of correct links, the lowest number of wrong links

World Statistics Day


Book of life
Book of life and applications.

Dunn (1946)* describes record linkage in this way:

…each person in the world creates a book of life. The book starts with the birth and ends with the death. Its pages are made up of all the principal events of life. Record linkage is the name given to the process of assembling the pages of this book into one volume. The person retains the same identity throughout the book. Except for advancing age, he is the same person…

*Dunn (1946) "Record Linkage". American Journal of Public Health 36 (12): 1412–1416.

World Statistics Day


When there is the lack of a unique identifier
When there is the lack of a unique identifier and applications.

If a record identifier is missing or cannot be used, it is necessary to use the common variables in the two files.

The problem is that these variables can be “unstable”:

  • Time changes (age, address, educational level)

  • Errors in data entry and coding

  • Correct answers but different codification (e.g. address)

  • Missing items

World Statistics Day


Main motivations for record linkage
Main motivations for record linkage and applications.

According to Fellegi (1997)*, the development of tools for integration is due to the intersection of these facts:

  • occasion: construction of big data bases

  • tool: computer

  • need: new informative needs

    *Fellegi (1997) “Record Linkage and Public Policy: A Dynamic Evolution”. In Alvey, Jamerson (eds) Record Linkage Techniques, Proceedings of an international workshop and exposition, Arlington (USA) 20-21 March 1997.

World Statistics Day


Why record linkage some examples
Why record linkage? Some examples and applications.

  • To have joint information on two or more variables observed in distinct data sources

  • To “enumerate” a population

  • To substitute (parts of) surveys with archives

  • To create a “list” of a population

  • Other official statistics objectives (imputation and editing / to enhance micro data quality; to study the risk of identification of the released micro data)

World Statistics Day


Example 1 analysis of mortality
Example 1 – analysis of mortality and applications.

Problem: to analyze jointly the “risk factors” with the event “death”.

  • The risk factors are observed on ad hoc surveys (e.g. those on nutrition habits, work conditions, etc.)

  • The event “death” (after some months the survey is conducted) can be taken from administrative archives

    These two sources (survey on the risk factors and death archive) should be “fused” so that each unit observed in the risk factor survey can be associated with a new dichotomous variable (equal to 1 if the person is dead and zero otherwise).

World Statistics Day


Example 2 to enumerate a population
Example 2 – to enumerate a population and applications.

Problem: what is the number of residents in Italy?

Often the number of residents is found in two steps, by means of a procedure known as “capture-recapture”. This method is usually applied to determine the size of animal populations.

  • Population census

  • Post enumeration survey (some months after the census) to evaluate Census quality and give an accurate estimate of the population size

    USA - in 1990 Post Enumeration Survey, in 2000 Accuracy and Coverage Evaluation

    Italy - in 2001 “Indagine di Copertura del Censimento”

World Statistics Day


Example 2 to enumerate a population1
Example 2 – to enumerate a population and applications.

The result of the comparison between Census and post enumeration survey is a 22 table:

World Statistics Day


Example 2 to enumerate a population2
Example 2 - to enumerate a population and applications.

For short, for any distinct unit it is necessary to understand if it was observed

1) both in the census and in the PES

2) only in the census

3) only in the PES

These three values allow to estimate (with an appropriate model) the fourth value.

World Statistics Day


Example 3 surveys and archives
Example 3 – surveys and archives and applications.

Problem: is it possible to use jointly administrative archives and sample surveys?

At the micro level this means: to modify the questionnaire of a survey dropping those questions that are already available on some administrative archives (reduction of the response burden)

E.g., for enterprises:

Social security archives, chambers of commerce, …

World Statistics Day


Example 4 creation of a list
Example 4 – Creation of a list and applications.

Problem: what is the set of the active enterprises in Italy?

In Istat, ASIA (Archivio Statistico delle Imprese Attive) is the most important example of a creation of a list of units (the active enterprises in a time instant) “fusing” different archives.

It is necessary to pay attention to:

  • Enterprises which are present in more than one archives (deduplication)

  • Non active enterprises

  • New born enterprises

  • transformations (that can lead to a new enterprise or to a continuation of the previous one)

World Statistics Day


Example 5 imputation and editing
Example 5 – Imputation and editing and applications.

Problem: to enhance microdata quality

Micro Integration in the Netherlands (virtual census, social statistical data base)

It will be seen later, when dealing with micro integration processing

World Statistics Day


Example 6 privacy
Example 6 - Privacy and applications.

Problem: does it exist a “measure” of the degree of identification of the released microdata?

In order to evaluate if a method for the protection of data disclosure is good, it is possible to compare two datasets (the true and the protected ones) and detect how many modified records are “easily” linked to the true ones.

World Statistics Day


Record linkage steps and applications.

The record linkage techniques are a multidisciplinary set of methods and practices

  • DECISION MODEL CHOICE

  • Fellegi & Sunter

  • exact

  • Knowledge – based

  • Mixed

  • SEARCH SPACE REDUCTION

  • Sorted Neighbourhood Method

  • Blocking

  • Hierarchical Grouping

......

RECORD

LINKAGE

......

......

  • PRE-PROCESSING

  • Conversion of upper/lower cases

  • Replacement of null strings

  • Standardization

  • Parsing

  • COMPARISON FUNCTION CHOICE

    • Edit distance

    • Smith-Waterman

    • Q-grams

    • Jaro string comparator

    • Soundex code

    • TF-IDF

World Statistics Day

Tiziana Tuoto, FCSM 2007, Arlington, November 6 2007


Example fortini 2008
Example (Fortini, 2008) and applications.*

Census is sometimes associated with a post enumeration surveys, in order to detect the actual census coverage.

To this purpose, a “capture-recapture” approach is generally considered.

It is necessary to find out how many individuals have been observed:

  • in both the census and the PES

  • Only in the census

  • Only in the PES

    These figures allow to estimate how many individuals have NOT been observed in both the census and the PES

    * In ESSnet Statistical Methodology Project on Integration of Survey and Administrative Data “Report of WP2. Recommendations on the use of methodologies for the integration of surveys and administrative data”, 2008

World Statistics Day


Record linkage workflow for census pes

CENSUS and applications.

PES

Matched households

Unmatched people

Unmatched people

Unmatched people

Unmatched households

Unmatched people

Unmatched households

Matched households

Matched people

Matched people

Matched people

Matched people

Matched people

Unmatched people

Record linkage workflow for Census - PES

Step 1

Step 2

Step 3.a

Step 3.b

Step 4.b

Step 4.a

Step 5

World Statistics Day


Problem lack of identifiers
Problem: Lack of identifiers and applications.

Difference between step 1 and step 2 is that:

Step 1 identifies all those households that coincide for all these variables:

  • Name, surname and date of birth of the household head

  • Address

  • Number of male and female components

    Step 2 uses the same keys, but admits the possibility of differences of the variable states for modifications of errors

World Statistics Day


Probabilistic record linkage
Probabilistic record linkage and applications.

For every pairs of records from the two data sets, it is necessary to estimate

  • The probability that the differences between what observed on the two records is due to chance, because the two records belong to the same unit

  • The probability that the two records belong to different units

    These probabilities are compared: this comparison is the basis for the decision whether a pair of records is a match or not

    Estimate of this probability is the “statistical step” in the probabilistic record linkage method

World Statistics Day


Statistical step
Statistical step and applications.

Data set A with na units.

Data set B with nb units.

K key variables (they jointly make an identifier)

World Statistics Day


Statistical procedure
Statistical procedure and applications.

The key variables of the two records in a pair (a,b) is compared:

yab=f(xAa,xBb)

The function f(.) should register how much the key variables observed in the two units are different.

For instance, y can be a vector with k components, composed of 0s (inequalities) or 1s (equalities)

The final result is a data set of na x nb comparisons

World Statistics Day


Statistical procedure1
Statistical procedure and applications.

The na x nb pairs are split in two sets:

M: the pairs that are a match

U: the unmatched pairs

Likely, the comparisons y will follow this situation:

  • Low levels of diversity for the pairs that are match, (a,b)M

  • High levels of diversity for the pairs that are non-match, (a,b)U

    For instance: if y=(sum of the equalities for the k key variables), y tends to assume large values for the pairs in M with respect to those in U

World Statistics Day


Statistical procedure2
Statistical procedure and applications.

If y=(sum of the equalities), the distribution of y is a mixture of the distribution of y in M (right) and that in u (left)

World Statistics Day


Statistical procedure3
Statistical procedure and applications.

Inclusion of a pair (a,b) in M or U is a missing value (latent variable).

Let C denote the status of a pair (C=1 if (a,b) in M; C=0 if (a,b) in U)

Likelihood is the product on the na x nb pairs of

P(Y=y, C=c) = [p m(y)]c [(1-p) u(y)](1-c)

Estimation method: maximum likelihood on a partially observed data set (EM algorithm – Expectation Maximization)

World Statistics Day


Statistical procedure4
Statistical procedure and applications.

A pair is assigned to M or U in the following way

1) For every comparison y assign a “weight”:

t(y)=m(y)/u(y)

where m and u are estimated;

2) Assign the pairs with a large weight to M and the pairs with a small weight to U.

3) There can be a class of weights t where it is better to avoid definitive decisions (m and u are similar)

World Statistics Day


Statistical procedure5
Statistical procedure and applications.

The procedure is the following.

Note that, generally, probabilities of mismatching are still not considered

World Statistics Day


Open problems
Open problems and applications.

Different probabilistic record linkage aspects should still be better investigated. Two of them are related to record linkage quality

  • What model should be considered

    • a1) on the pairs relationship (Copas and Hilton, 1990)

    • a2) on the key variables relationship (Thibaudeau, 1993)

      b) How probabilities of mismatching can be used for a statistical analysis of a linked data file? (Scheuren and Winkler, 1993, 1997)

      Copas J.R., Hilton F.J. (1990). “Record linkage: statistical models for matching computer records”. Journal of the Royal Statistical Society, Series A, 153, 287-320.

      Thibaudeau Y. (1993). “The discrimination power of dependency structures in record linkage”. Survey Methodology, 19, 31-38.

      Scheuren F., Winkler W.E. (1993). “Regression analysis of data files that are computer matched”. Survey Methodology, 19, 39-58

      Scheuren F., Winkler W.E. (1997). “Regression analysis of data files that are computer matched - part II”. Survey Methodology, 23, 157-165.

World Statistics Day


Statistical matching
Statistical matching and applications.

What kind of integration should be considered if the analysis involves two variables observed in two independent sample surveys?

  • Let A and B be two samples of size nA and nB respectively, drawn from the same population.

  • Some variables X are observed in both samples

  • Variables Y are observed only in A

  • Variables Z are observed only in B.

    Statistical matching aims at determining information on (X;Y;Z), or at least on the pairs of variables which are not observed jointly (Y;Z)

World Statistics Day


Statistical matching1
Statistical matching and applications.

It is very improbable that the two samples observe the same units, hence record linkage is useless.

World Statistics Day


Some statistical matching applications 1
Some statistical matching applications 1 and applications.

The objective of the integration of the Time Use Survey (TUS) and of the

Labour Force Survey (LFS) is to create at a micro level, a synthetic file of

both surveys that allows the study of the relationships between variables

measured in each specific survey.

By using together the data relative to the specific variables of both surveys,

one would be able to analyse the characteristics of employment and the

time balances at the same time.

Information on labour force units and the organisation of her/his life

times will help enhance the analyses of the labour market

The analyses of the working condition characteristics that result from

the labour force survey will integrate the TUS more general analysis of

the quality of life

World Statistics Day


Some statistical matching applications 11
Some statistical matching applications 1 and applications.

The possibilities for a reciprocal enrichment have been largely recognised

(see the 17th International Conference of Labour Statistics in 2003 and the

2003 and 2004 works of the Paris group). The emphasis was indeed put on

how the integration of the two surveys could contribute to analysing the

different participation modalities in the labour market determined by hour

and contract flexibility.

Among the issues raised by researchers on time use, we list the following

two:

the usefulness and limitations involved in using and combining various

sources, such as labour force and time-use surveys, for improving data

quality

Time-use surveys are useful, especially for measuring hours worked of

workers in the informal economy, in home-based work, and by the

hidden or undeclared workforce, as well as to measure absence from

work

World Statistics Day


Some statistical matching applications 12
Some statistical matching applications 1 and applications.

Specific variables in the TUS (Y ): it enables to estimate the time

dedicated to daily work and to study its level of "fragmentation" (number of intervals/interruptions), flexibility (exact start and end of working hours) and intra-relations with the other life times

Specific variables in the LFS (Z): The vastness of the information gathered allow us to examine the peculiar aspects of the Italian participation in the labour market: professional condition, economic activity sector, type of working hours, job duration, profession carried out, etc. Moreover, it is also possible to investigate dimensions relative to the quality of the job

World Statistics Day


Some statistical matching applications 2
Some statistical matching applications 2 and applications.

The Social Policy Simulation Database and Model (SPSD/M) is a micro computer-based product designed to assist those interested in analyzing the financial interactions of governments and individuals in Canada (see http://www.statcan.ca/english/spsd/spsdm.htm).

It can help one to assess the cost implications or income redistributive

effects of changes in the personal taxation and cash transfer system.

The SPSD is a non-confidential, statistically representative database of individuals in their family context, with enough information on each

individual to compute taxes paid to and cash transfers received from

government.

World Statistics Day


Some statistical matching applications 21
Some statistical matching applications 2 and applications.

The SPSM is a static accounting model which processes each individual and family on the SPSD, calculates taxes and transfers using legislated or proposed programs and algorithms, and reports on the results.

It gives the user a high degree of control over the inputs and outputs to the model and can allow the user to modify existing tax/transfer programs or test proposals for entirely new programs. The model can be run using a visual interface and it comes with full documentation.

World Statistics Day


Some statistical matching applications 22
Some statistical matching applications 2 and applications.

In order to apply the algorithms for microsimulation of tax–transfer benefits

policies, it is necessary to have a data set representative of the Canadian

population. This data set should contain information on structural (age,

sex,...), economic (income, house ownership, car ownership, ...), health–related (permanent illnesses, child care,...) social (elder assistance,

cultural–educational benefits,...) variables (among the others).

  • It does not exist a unique data set that contains all the variables that can influence the fiscal policy of a state

  • In Canada 4 samples are integrated (Survey of consumers finances, Tax return data, Unemployment insurance claim histories, Family expenditure survey)

  • Common variables: some socio-demographic variables

  • Interest is on the relation between the distinct variables in the different

    samples

World Statistics Day


Example coli et al 2006
Example (Coli et al, 2006*) and applications.

The new European System of the Accounts (ESA95) is a detailed source of information on all the economic agents, as households and enterprises. The social accounting matrix (SAM) has a relevant role.

Module on households: it includes the amount of expenditures and income, per typology of household

Coli A., Tartamella F., Sacco G., Faiella I., D’Orazio M., Di Zio M., Scanu M., Siciliani I., Colombini S., Masi A. (2006). “La costruzione di un Archivio di microdati sulle famiglie italiane ottenuto integrando l’indagine ISTAT sui consumi delle famiglie italiane e l’Indagine Banca d’Italia sui bilanci delle famiglie italiane”, Documenti ISTAT, n.12/2006.

World Statistics Day


Example
Example and applications.

Problem:

  • Income are observed on a Bank of Italy survey

  • Expenditures are observed on an Istat survey

  • The two samples are composed of different households, hence record linkage is useless

World Statistics Day


Adopted solutions 1
Adopted solutions 1 and applications.

The first statistical matching solution was imputation of missing data. Usually, “distance hot deck” was used.

In pratice, this method “mimics” record linkage: instead of matching records of the same unit, this approach “matches” records of similar units, where similarity is in terms of the common variables in the two files.

The procedure is

1) Compute the distances between the matching variables for every pair of records

2) Every record in A is associated to that record in B with minimum distance

World Statistics Day


Adopted solutions 11
Adopted solutions 1 and applications.

The inferential path is the following

World Statistics Day


Adopted solutions 2
Adopted solutions 2 and applications.

It is applied an estimate procedure under specific models that considers the presence of missing items. The easiest model is: conditional independence of the never jointly observed variables (e.g., income and expenditures) given the matching variables.

Example:

Y = income, Z = expenditures, X = house surface

(X,Y,Z) is distributed as a multivariate normal with parameters:

Mean vector = 

Variance matrix = 

World Statistics Day


Adopted solutions 21
Adopted solutions 2 and applications.

  • Estimate the regression equation on A: Y=+X

  • Impute Y in B: Yb=+Xb , b=1,…,nB

  • Estimate the regression equation in B: Z=+X

  • Impute Z in A: Za= +Xa , a=1,…,nA

World Statistics Day


Adopted solutions 22
Adopted solutions 2 and applications.

The inferential mechanism assumes that

Y and Z are independent given X

(there is not the regression coefficient of Z on Y

given X)

World Statistics Day


Adopted solutions 23
Adopted solutions 2 and applications.

This method can be applied also with this inferential scheme: the problem is what hypotheses are before the analysis phase

World Statistics Day


Adopted solutions 3
Adopted solutions 3 and applications.

We do not hypothesize any model. It is estimated a set of values, one for every plausible model given the observed data

Example

When matching two sample surveys on farms (Rica-Rea - FADN and SPA - FSS), it was asked the following contingency table for farms

Y = presence of cattle (FSS)

Z = class of intermediate consumption (from FADN)

Using the common variables

X1 = Utilized Agricultural Area (UAA) ,

X2 = Livestock Size Unit (LSU)

X3 = geographical characteristics

World Statistics Day


Example1
Example and applications.

We consider all the models that we cam estimate from the observed data in the two surveys

In practice, the available data allow to say that the estimate of the number of farms with at least one cow (Y=1) in the lowest class of intermediate consumption (Z=1) is between 2,9% and 4,9%

World Statistics Day


Inferential machine
Inferential machine and applications.

It is possible to simulate data including uncertainty on the data generation model (e.g. by multiple imputation)

The inferential machine does not use any specific model

World Statistics Day


Quotation manski 1995
Quotation (Manski, 1995*) and applications.

…”The pressure to produce answers, without qualifications, seems particularly intense in the environs of Washington, D.C. A perhaps apocryphal, but quite believable, story circulates about an economist’s attempt to describe his uncertainty about a forecast to President Lyndon Johnson. The economist presented his forecast as a likely range of values for the quantity under discussion. Johnson is said to have replied, “Ranges are for cattle. Give me a number”

*Manski, C. F. (1995) Identification problems in the Social Sciences, Harvard University Press.

Manski and other authors show that in a wide range of applied areas (econometrics, sociology, psychometrics) there is a problem of identifiability of the models of interest, usually caused by the presence of missing data. The statistical matching problem is an example of this.

World Statistics Day


Why statistical matching
Why statistical matching? and applications.

Applications in Istat

SAM

Joint analysis FADN / FSS

Joint use of Time Use / Labour force

Objectives

Estimates of parameters of not jointly observed parameters

Creation of synthetic data (e.g. data set for microsimulation)

World Statistics Day


Open problems1
Open problems and applications.

  • Uncertainty estimate (D’Orazio et al, 2006)

  • Variability of uncertainty (Imbens e Manski, 2004)

  • Use of sample drawn according to complex survey designs (Rubin, 1986; Renssen, 1998)

  • Use of nonparametric methods (Marella et al, 2008; Conti et al 2008)

    Conti P.L., Marella D., Scanu M. (2008). “Evaluation of matching noise for imputation techniques based on the local linear regression estimator”. Computational Statistics and Data Analysis, 53, 354-365.

    D’Orazio M., Di Zio M., Scanu M. (2006). “Statistical Matching for Categorical Data: Displaying Uncertainty and Using Logical Constraints”, Journal of Official Statistics, 22, 137-157.

    Imbens, G.W, Manski, C. F. (2004). "Confidence intervals for partially identified parameters". Econometrica, Vol. 72, No. 6 (November, 2004), 1845–1857 

    Marella D., Scanu M., Conti P.L. (2008). “On the matching noise of some nonparametric imputation procedures”, Statistics and Probability Letters, 78, 1593-1600.

    Renssen, R.H. (1998) Use of statistical matching techniques in calibration estimation. Survey Methodology 24, 171–183.

    Rubin, D.B. (1986) Statistical matching using file concatenation with adjusted weights and multiple imputations. Journal of Business and Economic Statistics 4, 87–94.

World Statistics Day


Micro integration processing
Micro integration processing and applications.

It can be applied every time it is produced a complete data set (micro level) by any kind of method. Up to now, applied after exact record linkage

Micro integration processing consists of putting in place all the necessary actions aimed to ensure better quality of the matched results as quality and timeliness of the matched files. It includes

  • defining checks,

  • editing procedures to get better estimates,

  • imputation procedures to get better estimates.

World Statistics Day


Micro integration processing1
Micro integration processing and applications.

It should be kept in mind that some sources are more reliable than others.

Some sources have a better coverage than others, and there may even be conflicting information between sources.

So, it is important to recognize the strong and weak points of all the data sources used.

World Statistics Day


Micro integration processing2
Micro integration processing and applications.

Since there are differences between sources, a micro integration process is needed to check data and adjust incorrect data. It is believed that integrated data will provide far more reliable results, because they are based on an optimal amount of information. Also the coverage of (sub) populations will be better, because when data are missing in one source, another source can be used. Another advantage of integration is that users of statistical information will get one figure on each social phenomenon, instead of a confusing number of different figures depending on which source has been used.

World Statistics Day


Micro integration processing3
Micro integration processing and applications.

During the micro integration of the data sources the following steps have to be taken (Van der Laan, 2000):

a. harmonisation of statistical units;

b. harmonisation of reference periods;

c. completion of populations (coverage);

d. harmonisation of variables, in case of differences in definition;

e. harmonisation of classifications;

f. adjustment for measurement errors, when corresponding variables still do not have the same value after harmonisation for differences in definitions;

g. imputations in the case of item nonresponse;

h. derivation of (new) variables; creation of variables out of different data sources;

i. checks for overall consistency.

All steps are controlled by a set of integration rules and fully automated.

World Statistics Day


Example micro integration processing
Example: Micro integration processing and applications.

From Schulte Nordholt, Linder (2007) Statistical Journal of the IAOS 24,163–171

Suppose that someone becomes unemployed at the end of November and gets unemployment benefits from the beginning of December. The jobs register may indicate that this person has lost the job at the end of the year, perhaps due to administrative delay or because of payments after job termination. The registration of benefits is believed to be more accurate. When confronting these facts the ’integrator’ could decide to change the date of termination of the job to the end of November, because it is unlikely that the person simultaneously had a job and benefits in December. Such decisions are made with the utmost care. As soon as there are convincing counter indications of other jobs register variables, indicating that the job was still there in December, the termination date will, in general, not be adjusted.

World Statistics Day


Example micro integration processing1
Example: Micro integration processing and applications.

Method: definition of rules for the creation of a usable complete data set after the linkage process.

If these approaches are not applied, the integrated data set can contain conflicting information at the micro level.

These approaches are still strictly based on quality of data sets knowledge.

Proposition for a possible next ESSnet on integration: study the links between imputation and editing activities and

World Statistics Day


Other supporting slides
Other supporting slides and applications.

World Statistics Day


Macro integration coherence of estimates
Macro integration: coherence of estimates and applications.

Sometimes it is useful to integrate aggregate data, where aggregates are computed from different sample surveys.

For instance: to include a set of tables in an information system

A problem is the coherence of information in different tables.

The adopted solution is at the estimate level: for instance, with calibration procedures (e.g.: the Virtual census in the Netherlands)

World Statistics Day


Project
Project and applications.

The objective of a project is to gather the developments in two distinct areas

Probabilistic expert systems: these are graphical models, characterized by the presence of an easy updating system of the joint distribution of a set of variables, once one of them is updated. These models have been used for a class of estimators that includes poststratification estimators

Statistical information systems: SIS for the production of statistical output (Istar) with the objective to integrate and manage statistical data given and validated by the Istat production areas, in order to produce purposeful output for the end users

World Statistics Day


Objectives and open problems
Objectives and open problems and applications.

Objectives

To develop a statistical information system for agriculture data, managing tables from FADN. FSS, and lists used for sampling (containing census and archive data)

To manage coherence bewteen different tables

To update information on data from the most recent survey and to visualize what changes happen to the other tables

To allow simulations (for policy making)

Problems

Use of graphical models for complex survey data

To link the selection of tables to the updating algorithm

To update more than one table at the same time

World Statistics Day


Some practical aspects for integration software
Some practical aspects for integration: Software and applications.

There exist different software tools for record linkage record linkage and statistical matching

Relais: http://www.istat.it/strumenti/metodi/software/analisi_dati/relais/

R package for statistical matching:

http://cran.r-project.org/index.html

Look for Statmatch

Probabilistic expert systems: Hugin (it does not work with complex survey data)

World Statistics Day


Bibliography
Bibliography and applications.

Batini C, Scannapieco M (2006) Data Quality, Springer Verlag, Heidelberg.

Scanu M (2003) Metodi statistici per il record linkage, collana Metodi e Norme n. 16, Istat.

D’Orazio M., Di Zio M., Scanu M. (2006) Statistical matching: theory and practice, J. Wiley & Sons, Chichester.

Ballin M., De Francisci S., Scanu M., Tininini L., Vicard P. (2009) Integrated statistical systems: an approach to preserve coherence between a set of surveys based on the use of probabilistic expert systems, NTTS 2009, Bruxelles.

World Statistics Day


Is this conditional independence
Is this conditional independence? and applications.

World Statistics Day


And this
And this? and applications.

World Statistics Day


Statistical methods of integration2
Statistical methods of integration and applications.

Sometimes a “shorter track” is used.

Note! The “automatic methods” correspond to specific data generating model

World Statistics Day


Statistical methods of integration3
Statistical methods of integration and applications.

World Statistics Day


Statistical methods of integration4
Statistical methods of integration and applications.

The last approach is very appealing:

  • Estimate a data generating model from the two data samples at hand

  • Use this estimate for the estimation of aggregate data (e.g. contingency tables on non jointly observed variables)

  • If necessary, develop a complete data set by simulation from the estimated model: the integrated data generating mechanism is the “nearest” to the data generating model, according to the optimality properties of the model estimator

    Attention! Issue 1 includes hypothesis that cannot be tested on the available data (this is true for record linkage and, more “dramatically”, for statistical matching)

World Statistics Day


ad