- 115 Views
- Uploaded on
- Presentation posted in: General

Using Bootstrap in NLSCY

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Today’s Presentation

B O O T S T R A P

- We’ll discuss the guiding principles
- We’ll demonstrate the CV lookup spreadsheet (which is based on thebootstrap weights)

- Bootstrap macros by example
- Summarize some technical aspects.

The National Longitudinal Survey of Children and Youth measures a wide array of characteristics related to child and youth development

There are many opportunities for statistical inference

The number of possibilities is further compounded by the longitudinal character of the survey

A basic problem of inference is finding the variability of the estimators.

- Does not need exact formulas.
- Takes into account design information.
- It can be adapted to the desired level of precision.
- Computer intensive

A) Take a subsample of the original sample - trying to mimic the initial selection process.

B) For this subsample compute weights as if it was the actual sample. The result is a bootstrap weight.

Repeat A) and B) many times to obtain a set of bootstrap weights

Note that both A) and B) make essential use of the design information

Now suppose we are interested in an estimate;

- Compute the estimate using each of the bootstrap weights

- Compute the variance of the obtained points.

Note: These two steps are implemented in any program or software that uses bootstrap weights to assess sampling variability.

Using the release weights gives the correct estimates.

However, the variance of the estimator provided by SAS or SPSS is not the real one - most of the time it is less.

Here are the comparisons for two examples:

Average

Regression Coefficients

- We have two tools at hand:
- A database of variances for proportions - already computed by bootstrap
- The bootstrap macros

For the variability of proportion estimates we can use an Excel table of already computed results. The work has been done using bootstrap.

This table replaces the usual look-up tables for variance.

One can choose the domain based on age and province.

This general framework allows for estimating variability of proportions in future cycles of the survey.

In most situations where proportions are involved, consulting this database may be enough.

Here are examples on how to use it.

skip

Question: What is the quality (c.v) of the estimates for the proportion of girls aged 3 in Newfoundland at cycle 3?

How many will there be in cycle 5?

Will the quality suffer from the smaller sample size?

Click on the right arrow in

Province to select a province

intro/skip

Select Newfoundland (Terre-Neuve)

Click to select C3 Age = 3

Since the proportion of girls should be

around 50%, click on Prop. Cible and

select 50%.

The remaining rows contain the results that interest us ...

You can now see that the c.v. for that particular domain in

cycle 3 was 17.5% with 44 children in the sample.

In cycle 5, we predict 35 children will be left in sample

(assuming 90% response rate in cycle 4 and 5) and the c.v.

will grow to 19.6%.

intro/sauter

Question: What domains based on a 15% proportion are not publishable?

We are looking for domains with a c.v. higher

than 33.33%

Click and select Prop. cible of 15%

Click and select Custom in bs_cv

Select “is greater than” in the first field

Finally, type in 33.33 in the second field

and click OK

You can now see the first few rows of

estimates that we can’t release according to

customary quality level guidelines.

The table contains variance estimations obtained by bootstrap - under general conditions.

It is best suited for quick general assessment of variance and projections for future cycles.

When we need the most accurate variance estimation we have to do the bootstrap for the specific variable of interest.

intro

- Bootstrap weights are computed and made available by methodology.
- The user runs the macros.

- Preparing the input
- Specifying the options and running the macros
- Saving and interpreting the results

- Two input files are required:
- The bootstrap weights file
- The file with variables of interest

- These files must be merged - ususally with the CHILDID identifier

The options to be specified are as follows:

(i) The kind of estimator.

(ii) Whether the analysis is done globally or by domains.

(iii) SAS libraries.

(iv) The names of the variables for analysis.

(v) The number of bootstrap weights to be used.

(i) The built-in choices - in the current version - are:

Other estimators may require customizing the code

(ii) If analysis by subgroup is desired, the user needs to specify the subgroup variable.

Totals

Linear Regression

Ratios

Logistic Regression

Difference of Ratios

a) Estimate variance of a total by region

b) Estimate the variance of an average

c) Estimate the variance of regression coefficients

Problem:

Find the variance of the total number of bedrooms in

households with teenagers within each province - as

estimated from the sample.

/*

%partition(domains=); *no partition if no variable name provided;

%total(dataset=,variable=,nb_weights=);

COLLECT OUTPUT FROM DATASET: totals

%ratio(dataset=,numerator=,denominator=,nb_weights=);

COLLECT OUTPUT FROM DATASET: ratios - in PERCENTS -

%ratio_difference(dataset=,numerator1=,denominator1=,

numerator2=,denominator2=,nb_weights=);

COLLECT OUTPUT FROM DATASET: diffrat - in PERCENTS -

%regression (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reg

%logistic_reg (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reglg

NOTE: unless explicitly deleted, the datasets mentioned above

will keep accumulating the results of successive macro calls */

/*

%partition(domains=); *no partition if no variable name provided;

/*

%partition(domains=); *no partition if no variable name provided;

%total(dataset=,variable=,nb_weights=);

COLLECT OUTPUT FROM DATASET: totals

%ratio(dataset=,numerator=,denominator=,nb_weights=);

COLLECT OUTPUT FROM DATASET: ratios - in PERCENTS -

%ratio_difference(dataset=,numerator1=,denominator1=,

numerator2=,denominator2=,nb_weights=);

COLLECT OUTPUT FROM DATASET: diffrat - in PERCENTS -

%regression (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reg

%logistic_reg (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reglg

NOTE: unless explicitly deleted, the datasets mentioned above

will keep accumulating the results of successive macro calls */

%total(dataset=,variable=,nb_weights=);

COLLECT OUTPUT FROM DATASET: totals

%include

"C:\users\dochcat\bootstrap\NLSCY_VES.sas";

%let weight_path = C:\users\dochcat\bootstrap\Bs_Weights;

%let weights = bvar;

libname wt_lib "&weight_path";

%let data_path = C:\users\dochcat\Data;

%let data = basic_set;

libname dt_lib "&data_path";

%let save_path = C:\users\dochcat\bootstrap\Results;

%let output = table01;

libname sv_lib "&save_path";

procsortdata=wt_lib.&weights out=weights;

by childid; run;

procsortdata=dt_lib.&data (where=(cmmcq01>12))

/*keep only teenagers*/ out=dataset;

by childid; run;

data data_and_weights;

merge dataset(in=a) weights(in=b);

by childid;

if a; * keep only the necessary records;

run;

/* initialise totals */

procdatasetslibrary=work; delete totals; run;

%partition(domains=cgehd03);

%total(dataset=data_and_weights,

variable=nb_bedrooms,

nb_weights=1000);

/*save results*/

datasv_lib.&output; set totals; run;

procprintdata=sv_lib.table01; run;

back

back

Problem:

For children of age 6, find the average number of years of

education of the Person Most Knowledgeable about the child

Note:

Even though the average was not mentioned as an available

option, it is easily computed as a ratio.

/*

%partition(domains=); *no partition if no variable name provided;

%total(dataset=,variable=,nb_weights=);

COLLECT OUTPUT FROM DATASET: totals

%ratio(dataset=,numerator=,denominator=,nb_weights=);

COLLECT OUTPUT FROM DATASET: ratios - in PERCENTS -

%ratio_difference(dataset=,numerator1=,denominator1=,

numerator2=,denominator2=,nb_weights=);

COLLECT OUTPUT FROM DATASET: diffrat - in PERCENTS -

%regression (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reg

%logistic_reg (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reglg

NOTE: unless explicitly deleted, the datasets mentioned above

will keep accumulating the results of successive macro calls */

%ratio(dataset=,numerator=,denominator=,nb_weights=);

COLLECT OUTPUT FROM DATASET: ratios - in PERCENTS -

%include

"C:\users\dochcat\bootstrap\NLSCY_VES.sas";

%let weight_path = C:\users\dochcat\bootstrap\Bs_Weights;

%let weights = bvar;

libname wt_lib "&weight_path";

%let data_path = C:\users\dochcat\Data;

%let data = basic_set;

libname dt_lib "&data_path";

%let save_path = C:\users\dochcat\bootstrap\Results;

%let output = table02;

libname sv_lib "&save_path";

procsortdata=wt_lib.&weights out=weights;

by childid; run;

procsortdata=dt_lib.&data

(where=(cmmcq01=6 and cedpd04<96))

/*keep only age 6 kids with valid values

of the variable*/

out=dataset;by childid; run;

data data_and_weights;

merge dataset(in=a) weights(in=b);

by childid;

if a; *keep only the necessary records;

count=1; *necessary for average calculation;

run;

/* initialise ratios */

procdatasetslibrary=work; delete ratios; run;

%partition(domains=); *ensure no partition;

%ratio(dataset=data_and_weights,

numerator=cedpd04,

denominator=count,

nb_weights=1000);

/* save the results */

datasv_lib.&output; set ratios; run;

procprintdata=sv_lib.table02; run;

Note that the result is expressed as a percentage. For our purposes we need to divide the standard errors, confidence limits, and estimates by 100, and the variance by 100*100. The coefficient of variance stays the same.

Hence the results are: Mean=12.8446, with a 95% confidence interval [12.6544 , 13.0348]

Let us compare the confidence interval we just computed with the confidence interval produced by using only the release weights.

The following SAS code will produce these confidence limits:

procmeansmeanlclmuclmdata=data_and_weights;

var cedpd04;

weight w_final;

run;

And they are …

… while the bootstrap estimate of the same confidence interval is:

[12.6544 , 13.0348]

This is what we get when we compare:

Bootstrap

Classical

We can see an increase by a factor of about 1.7 - for this variable.

back /intro/skip

Problem:

Estimate the variance of regression coefficients of an outcome variable - the PPVT score. The independent variables are: the number of years of education of the PMK, and positive interaction in parenting.

/*

%partition(domains=); *no partition if no variable name provided;

%total(dataset=,variable=,nb_weights=);

COLLECT OUTPUT FROM DATASET: totals

%ratio(dataset=,numerator=,denominator=,nb_weights=);

COLLECT OUTPUT FROM DATASET: ratios - in PERCENTS -

%ratio_difference(dataset=,numerator1=,denominator1=,

numerator2=,denominator2=,nb_weights=);

COLLECT OUTPUT FROM DATASET: diffrat - in PERCENTS -

%regression (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reg

%logistic_reg (dataset=,dependent=,independent=,nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reglg

NOTE: unless explicitly deleted, the datasets mentioned above

will keep accumulating the results of successive macro calls */

%regression (dataset=,dependent=,independent=, nb_weights=);

COLLECT OUTPUT FROM DATASET: bs_reg

%include

"C:\users\dochcat\bootstrap\NLSCY_VES.sas";

%let weight_path = C:\users\dochcat\bootstrap\Bs_Weights;

%let weights = bvar;

libname wt_lib "&weight_path";

%let data_path = C:\users\dochcat\Data;

%let data = basic_set;

libname dt_lib "&data_path";

%let save_path = C:\users\dochcat\bootstrap\Results;

%let output = table03;

libname sv_lib "&save_path";

procsortdata=wt_lib.&weights out=weights;

by childid; run;

procsortdata=dt_lib.&data

(where=(cedpd04<96 and cprcs03<96

and cppcs01<888 and cmmcq01 in (6))) out=dataset;

/*always eliminate records with nonresponse codes*/

by childid; run;

data data_and_weights;

merge dataset(in=a) weights;

by childid;

if a; *keep only the necessary records;

run;

Estimate variance of regression coefficients - continued

%start_chronometer;

/* initialise bs_reg */

procdatasetslibrary=work; delete bs_reg; run;

%partition(domains=);

%regression(dataset=data_and_weights,

dependent=cppcs01,

independent=cprcs03 cedpd04,

nb_weights=100);

/* save the results */

datasv_lib.&output; set bs_reg; run;

%stop_chronometer;

Now we compare the bootstrap coefficients of variation for the regression parameters with the coefficients of variation produced by the SAS regression procedure.

procregdata=data_and_weights;

model cppcs01 = cprcs03 cedpd04;

weight an_weight;

run;

quit;

Here is the SAS output:

From the estimates and their standard errors we compute the CV’s and compare:

One can see that the results are very different. In such a situation the SAS results are not to be trusted

back/intro

- Generally the running time is equal to the number of bootstrap weights used multiplied by the time it takes for one estimate. The most time consuming are the bootstraps for regression or logistic regression.
- As an example, consider a Pentium II 350 Mhz with 128MB memory. On this machine we ran bootstraps for the variance of regression coefficients with 1,000 weights. The running time was usually 45 to 60 minutes.
- Since we are dealing with large files, the computing performance will greatly benefit from large amounts of RAM.

Two simple macros are provided in our file for measuring the running time. They are used as follows:

…….

%start_chronometer;

%partition(domains=cgehd03);

%total(dataset=data_and_weights,variable=nb_bedrooms,

nb_weights=1000);

%stop_chronometer;

…….

After the call of the %stop_chronometer macro, the time elapsed - in seconds - is written to the SAS log

- Averages are ratios. The user has to set up the necessary variables - see the case of the variable “count” in the examples.

- Generally, curious users may find useful tricks for their programs inside the NLSCY_VES.sas file. Of course, modifying the file itself will make it harder to benefit from the experience of other users.

skip

- The results accumulate in the output dataset when several macros are run in sequence. Only when the respective dataset is deleted explicitly the previous numbers are lost.

- The macros use temporary data sets. Conflicts of names with user defined datasets are possible. If you suspect this is the case inspect the log for all the datasets that are being created or deleted.

- The point estimates provided in the results are computed based on the release weights. Only their variances are computed by using the bootstrap weights. The reason for providing the estimates along with the respective variances is to allow the users to double check their work.

- Sometimes the macros have to output a lot of text to the log window. If you plan to run unsupervised jobs it is advisable to redirect the log output to a file.

- When testing, it is good to keep in mind that some steps may not need to be executed again. For instance, once the “data_and_weights” are in memory, only the macros and their auxiliary statements are needed. Also, specifying a smaller number of weights may greatly reduce the testing time.

- The macros presented here allow for an easy utilization of the bootstrap weights. In most situations the user only needs to write code pertaining to his/her dataset, and not to the bootstrap process.
- When more complex estimators are required, the user may need to write custom code. The code of the macros presented here can serve as a template.