Adult population survey aps training session
This presentation is the property of its rightful owner.
Sponsored Links
1 / 68

ADULT POPULATION SURVEY (APS) TRAINING SESSION PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on
  • Presentation posted in: General

ADULT POPULATION SURVEY (APS) TRAINING SESSION. Yana Litovsky GEM Data Team Supervisor Jonathan Francis Carmona GEM Data Manager. This session will cover:. APS tools & guidelines APS Questionnaire Survey Design Weights Sample Size Step in APS Cycle: Proposal Submission

Download Presentation

ADULT POPULATION SURVEY (APS) TRAINING SESSION

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Adult population survey aps training session

ADULT POPULATION SURVEY (APS) TRAINING SESSION

Yana Litovsky

GEM Data Team Supervisor

Jonathan Francis Carmona

GEM Data Manager


This session will cover

This session will cover:

APS tools & guidelines

  • APS Questionnaire

  • Survey Design

  • Weights

  • Sample Size

    Step in APS Cycle:

  • Proposal Submission

  • Data Collection

  • Data Review

  • Processed Results


Some aps facts

  • the Adult Population Survey (APS) investigates the role of the individual in the lifecycle of the entrepreneurial process and his or her characteristics or actions

  • a representative sample of the adult population is used to estimate the entrepreneurial activity that represents the entire country.

  • The APS is unique because it measures

    • Motives

    • Attitudes

    • Actions

    • Business characteristics

Some APS Facts


Gem entrepreneurial phases

GEM Entrepreneurial Phases


Aps questionnaire

APS Questionnaire


Aps questionnaire1

  • The APS is comprised of MODULES of different groups of questions, often on different topics

  • The modules are distributed within the questionnaire in different BLOCKSfor different target respondents

APS Questionnaire


Aps blocks

Introduction: All respondents

Block 1: Nascent entrepreneurs

Block 2: Owner-managers

Block 3: Potential and discontinuing

Block 4: Informal investors

Block 5: All respondents (occupation)

Block 6: Reserved (not used in 2014)

Block 7: All respondents (demographics)

APS Blocks


Aps modules

  • Core APS Module

    • Remain consistent year to year

    • Used to derive main GEM indicators

    • Must be included by all GEM National Teams

  • Special Topic Module

    • Set of questions that change year to year

    • On different research topics

    • May be included in one or more blocks

    • May include optional questions

  • Optional Modules

    • Questions on different topics

    • Suggested by national teams or derived from previous Special Topics

    • Follow placement instructions

APS Modules


Block 1 nascent entrepreneurs flow

Block 1: Nascent entrepreneurs (flow)

INTRODUCTION

REQUIRED QUESTIONS FOR ENTIRE SAMPLE

OPTIONAL QUESTIONS FOR ENTIRE SAMPLE

FILTER QUESTIONS FOR ALL RESPONDENTS

BLOCK1

QUALIFIED RESPONDENTS

REQUIRED QUESTIONS FOR NASCENT ENTREPRENEURS

SPECIAL TOPIC: REQUIRED QUESTIONS FOR NASCENT ENTREPRENEURS

UN-QUALIFIED RESPONDENTS

OPTIONAL QUESTIONS FOR NASCENT ENTREPRENEURS

BLOCK2


Block 2 owner managers flow

Block 2: Owner-managers (flow)

FILTER QUESTIONS FOR ALL RESPONDENTS

BLOCK2

QUALIFIED RESPONDENTS

REQUIRED QUESTIONS FOR OWNER MANAGERS

SPECIAL TOPIC: REQUIRED QUESTIONS FOR OWNER MANAGERS

UN-QUALIFIED RESPONDENTS

OPTIONAL QUESTIONS FOR OWNER MANAGERS

BLOCK3


Blocks 3 4

Blocks 3 & 4

FILTER QUESTIONS FOR ALL RESPONDENTS

BLOCK3

REQUIRED QUESTIONS FOR POTENTIAL & DISCONTINUING

QUALIFIED RESPONDENTS

OPTIONAL QUESTIONS FOR POTENTIAL & DISCONTINUING

UN-QUALIFIED RESPONDENTS

BLOCK4

REQUIRED QUESTIONS FOR INFORMAL INVESTORS

OPTIONAL QUESTIONS FOR INFORMAL INVESTORS


Blocks 5 7

Blocks 5 & 7

BLOCK5

REQUIRED QUESTIONS FOR ALL (OCCUPATION)

SPECIAL TOPIC: REQUIRED QUESTIONS FOR ALL (OR BASED ON OCCUPATION STATUS)

OPTIONAL QUESTIONS FOR ALL (OR BASED ON OCCUPATION STATUS)

BLOCK7

REQUIRED QUESTIONS FOR ALL (DEMOGRAPHICS)

REQUIRED VENDOR PROVIDED INFORMATOIN

OPTIONAL QUESTIONS FOR ALL


Aps questionnaire format

Full questionnaire, with codes and skip patterns.

APS Questionnaire Format


Aps questionnaire format1

Each block is color-coded

APS Questionnaire Format


Survey design

Survey Design


Survey design guidelines

  • There is no one standard GEM methodology!

  • GEM’s only fixed requirement is that a team collects a nationally representative sample of at least 2000 adults

  • To ensure this, GEM provides guidelines about:

    • Proper choice of respondent contact method

    • Sufficient number of callbacks and initial contact attempts

    • Appropriate time of survey administration

    • Appropriate stratification of sample

      Some guidelines stable but others differ from team to team

Survey Design Guidelines


Contact method options

  • Sample can be using any combination of:

    • Face-to-face

    • Fixed line

    • Mobile phone

    • Choice based on feasibility, budget & bias

  • Each methodology can have its own bias, for example:

    • Fixed-line usually biased toward older respondents

    • Mobile phone usually biased toward younger respondents

    • Face-to-face may be biased against people often out of the house

Contact Method Options


Sample design options

Most teams collect a sample that is proportional to the national population distribution

Others may add an oversample: may collect disproportionately more responses in a metropolitan area

Required to meet the n=2000 minimum not counting any oversamples.

Oversample may be part of Global Report sample or submitted separately

Sample Design Options


Weights

Weights


Weights1

All analysis of GEM APS data MUST use weights.

National weight mean =1 (do not account for different sample sizes).

Weights differ by the type and age/range of sample.

Weighted age/gender distribution must be nationally representative.

Strata-based weights must match overall strata sizes.

Very large values for weights (high leverage cases) are not allowed.

Weights


Types of weights

  • RESPONSE WEIGHTS: Used to achieve a better match of the sample surveyed to that of the overall population.

    • To reduce potential bias due to different responses rates or normal sampling variability.

    • Required for every country.

  • SAMPLE DESIGN WEIGHTS: Used to adjust a sample that was purposefully designed to have unequal representation

    • For example, if a team oversamples (by design) in a metropolitan area, weights are used to not over-represent the city respondents.

Types of Weights


Gem national aps weights

  • WEIGHT_L

    • Used for the adult population aged 18 to 64 (L = Labor Force).

  • WEIGHT_A

    • Used for the entire adult population aged 18 to 99 (A = Adults)

  • If national team only collected data for ages 18 to 64, then Weight_L and Weight_A are identical.

GEM National APS Weights


Gem regional aps weights

  • Only provided for teams who request regional-level analysis.

    • WEIGHT_LR

    • WEIGHT_AR

  • Each region has its own weights calculated as if they were the entire nation.

  • Should be used ONLY for region-to-region comparisons, not for any national-level analysis!

GEM Regional APS Weights


Final lesson

Always use weights for any GEM APS individual-level analysis.

If you want to find comparable Global Report results, use WEIGHT_L

Final Lesson


Sample size

Sample Size


Sample sizes 2013

Sample Sizes (2013)

Minimum sample

N = 2000

Appropriate regardless of national population size


Respondents per block fraction of sample

respondents per block (fraction of sample)


Respondents per block number in 2000 person samples

respondents per block(number in 2000 person samples)


Sample size combining individual level data

But sample sizes vary by nation and year.

Beware when combining data from nations or years with different sample sizes.

Weights do NOT adjust for different sized samples

The weighted total for any national analysis will match the unweighted total (that is why weights have a mean of 1).

Sample Size:combining individual level data


Proposal submission process

Proposal Submission Process


Rfp package

Request for Proposal (RFP) is posted to the GEM consortium website and an announcement is emailed to all teams in February of each year.

The RFP includes all documents needed by National Team to submit APS/NES proposal, administer the questionnaires, record and submit all data. It also includes valuable reference information.

“RFP” refers to the entire document package as well as the Request for Proposal document itself.

RFP Package


Rfp package contents

  • Data Delivery Documents:

    • SPSS Data Input Templates (APS & NES)

  • Proposal Documents:

    • Instructions for submitting proposals (APS & NES)

    • Survey Reports (APS & NES)

    • Additional NES instructional documents

  • Questionnaire Documents:

    • Final version of questionnaires (APS & NES; English & Spanish)

    • APS question-by-question commentary

  • Reference Documents:

    • PowerPoint Training presentation (APS & NES)

    • GEM Timeline

    • Reference Information (additional APS questions, education categories, optional module)

RFP Package (contents)


Rfp package data delivery spss data input file

Use provided SPSS Data Input Template to submit final survey data.

Data in any other format will not be processed

RFP PackageData Delivery: SPSS data input file


Rfp package questionnaire aps commentary

Detailed description of each question

RFP PackageQuestionnaire: APS commentary


Rfp package proposal aps survey report

~ Excel document comprised of several worksheets

~ Unique for each team

  • Methodology Overview

  • Previous year’s Data Quality Report

  • APS Questionnaire information

  • Sample strata description & fieldwork report

  • Population statistics

  • Weight construction

  • Education categories

  • Additional worksheets populated during APS cycle

RFP PackageProposal: APS Survey Report


Rfp package proposal aps survey report1

Worksheet A: Methodology Overview

  • Provide full details of sample design (usually completed by vendor)

  • Information included in Global Report

RFP PackageProposal: APS Survey Report

  • Sample size

  • Interview method

  • Sampling methodology

  • Sample stratification

  • Call backs

  • Previous data issues


Rfp package proposal aps survey report2

Worksheet C: Strata & Fieldwork Report

  • List and describe sample strata (if none used, national information)

  • Complete fieldwork report for each strata after data is collected

RFP PackageProposal: APS Survey Report


Preparing an aps p roposal

  • Download and carefully read the RFP package

  • Work with Survey Vendor to decide on appropriate survey methodology and fill out all required RFP documents.

    • Vendors must use GEM forms except for Vendor Proposal(has no template)

  • Customize your questionnaire and SPSS Data Input Template

    • Decide which optional modules you will include (if any)

    • Decide if you will add additional questions (must be approved by GEM)

    • Establish you demographic variable answer categories

    • Translate the questionnaire if necessary

      Final questionnaire is each team’s responsibility

Preparing an APS Proposal


Tips for translating aps questionnaire

  • Work with vendor on translations

    • National team has more knowledge of question purpose

    • Vendor can advise how questions will be received by respondent

    • Use question-by-question commentary

    • Always do a back-translation

    • Can share other team’s translations with permission

Tips for translating APS questionnaire


Tips for education categories

~ Team creates own categories for education variable

~ GEM recodes categories into harmonized variables

Keep our recoding rules in mind when constructing your education variable

  • Respondent must complete an education level (e.g. earn a degree) to be coded in that category

  • Bachelors, Masters and Doctorate degrees are best kept in separate categories

Tips for EDUCATION categories


Tips for income categories

  • Team creates their own categories for income variable

  • GEM requires at least 6 income ranges

  • GEM recodes variable into lower, middle, upper percentile

    Keep national income distribution in mind! If most respondents fall into one income range, this variable is not useful.

Tips for INCOME categories


Submitting an aps proposal

  • Email the completed forms to [email protected] (refer to RFP for instructions on how to label documents).

  • The GEM Data team will review submitted documents and proposed sampling methodology and send National Team a review in a new worksheet added to Survey Report document (G. Proposal Review).

  • The national team MAY NOT BEGIN APS survey until their proposal has been officially approved (via email) by GEM Data Team.

Submitting an APS Proposal


Evaluation of aps proposal

  • Proposal document review: has all the required material been submitted?

    • Every mandatory chart applicable to your proposal must be completed

    • All information must be submitted in required format

  • APS Questionnaire review: do teamquestions interfere with core APS?

    • Content and placement of any additional APS questions must be approved by Data Team

    • Verbal introduction of the survey to a potential respondent are reviewed(see RFP tips)

  • Methodology review: does sample/survey design meet GEM standards?

    • Is sample stratification appropriate?

    • Is the timing of survey administration appropriate?

    • Is a sufficient number of call-backs and initial contact attempts proposed?

    • Have all data issues from previous years been addressed?

Evaluation of APS Proposal


Evaluation of aps proposal1

If necessary, the National Team will be requested to:

submitany missing information

clarify specifics of the sample design

revisesampling and survey plan

Evaluation of APS Proposal


Adult population survey aps training session

DO NOT WAIT FOR PROPOSAL TO BE COMPLETE BEFORE GETTING IN TOUCH WITH GEM DATA TEAM IF YOU HAVE QUESTIONS

~ We can give suggestions about choice of vendor

~ Pre-evaluate different methodologies

~ Help team conduct a pilot test


Data collection process

Data Collection Process


Before aps data collection

Mentor Team: New teams are recommended to find a find a mentor GEM country. Contact other GEM teams with same cultural background, especially if they have the same language.

Before APS Data Collection


Before aps data collection1

  • Pilot Study: New team or teams with new vendor will be required to conduct pilot study with a minimum of 100 respondents and submit results before continuing data collection.

    • Results (submitted using APS Data Input Template) will be tested for:

      • proper skip pattern

      • correct variable coding

      • proper use of data templates

      • no excessive missing data

  • Any problems will be quickly communicated

  • If no problems, pilot respondents can be included in final sample.

  • A team must guarantee that any will be addressed and corrected

Before APS Data Collection


During aps data collection how to check your data

Interim Data Submission: If Pilot Study is not required, teams are still encouraged to submit interim data file to be checked for data format, skip logic or other errors.

Data Quality Syntax: A national team can use SPSS syntax to check their data before submitting to GEM. These files replicate most GEM data checks.

During APS Data Collection: How to check your data


During aps data collection errors to check and avoid

  • Skip logic errors due to incorrect questionnaire programing or poorly trained interviewers: If these result in excessive missing data, team may be asked to recontact participants (if possible) or resample, in order for the data to be approved and included in the Global Report (extra data is also a problem)

During APS Data Collection: Errors to check and avoid


During aps data collection errors to check and avoid1

  • Unrepresentative sample due to incorrectly implemented sampling procedure: If final sample does not reflect the age/gender distribution of country within an acceptable range, team’s data may not be acceptable.

During APS Data Collection: Errors to check and avoid


Recording aps data

Some basic guidelines:

  • All recorded data must correspond to a designated answer category

  • Each respondent ID is unique and no greater than 9 digits

  • Do not delete variables from SPSS template

  • Do not use any digits or symbols to indicate a blank cell (e.g. “0”)

  • Add value labels for demographics with team-created categories

    ... see all guidelines in RFP and Data Quality Syntax

Recording APS Data


Open ended text responses

What kind of business is this?

  • Response must describe:

    • The nature of the product or service as well as

    • The primary customer base: other businesses or typical consumers

    • The detailed sector of activity

  • Common businesses need little elaboration:

    • Restaurant, homebuilder, day care center, hair salon, etc.

Open-ended Text Responses


Common problematic responses

Common problematic responses


Verify survey report information

Confirm that all Survey Report information submitted before data collection began is still correct:

  • Education categories are the identical in Survey Report and dataset

  • Population data used to calculate weights same as provided in Survey Report

  • Strata listed in Survey Report have the same names, value labels order

  • Correct weight formula was indicated

    If any changes must be made to Survey Report, make the necessary revisions and alert GEM Data Team when revised document is submitted.

Verify Survey Report Information


Submitting aps dataset

  • Submit APS Data in SPSS template (follow document naming conventions indicated in RFP)

  • Complete Strata & Fieldwork Report in Survey Report and submit with APS Data

    • Number of answered calls/doors

    • Number of eligible households contacted

    • Number of interviews agreed to...

  • Send both documents to [email protected]

  • Submitting APS Dataset


    Data review process

    Data Review Process


    Initial review of aps data

    Initial data checks

    1. Data is correctly recorded and formatted

    2. Unique, numeric IDs value for each respondent

    3. Proper variable and value labels are used

    4. All mandatory questions have been asked

    Weights

    1. Representative of age/gender distribution in country and, if sample is stratified, representative of

    strata distribution.

    2. No excessive number of high value weights

    Open-ended responses

    1. Translated into English

    2. Both native and English language responses provided in appropriate variables.

    Skip logic checks

    1. No excessive missing data

    2. If data missing, how critical are the errors?

    Initial Review of APS Data


    Data review sent to national team

    Example of APS Data Review Issues (worksheet H. Data Review, in Survey Report)

    Data Review (sent to national team)


    Data quality report

    • Comprehensive review of data quality and team performance

    • Included in new worksheet in Survey Report (I. Data Quality Report)

    • Includes:

      • % missing for key variables

      • % refused for key variables

      • % skip logic errors

      • Key indicators for past 3 years

      • Sample distribution

      • Deadlines

      • Exchange rate

      • Proposal/APS Issues

    Data Quality Report


    Data quality report1

    Data Quality Report


    Aps processed results

    APS Processed Results


    Aps data processing levels

    APS Data Processing Levels


    Computing aps measures

    • Data Team uses cleaned, harmonized and merged datasets to compute indices from core APS data.

    • Some individual level data variable names and variables differ from merged national level data.

    • Variable codebooks for both individual and national level data made available with processed APS results.

    • National level results (weighted by WEIGHT_L) are reported in the Global Report.

    Computing APS measures


    Tea creation flowchart

    TEA Creation Flowchart


    What data is available

    What Data is Available?


    Timeline

    February

    RFP released

    mid-March

    NES RFP due

    late-April

    APS RFP (Survey Report and Survey Vendor Proposal) due

    mid-July

    APS & NES data and documentation (including Fieldwork Report) due

    mid to late-September

    First release of APS & NES master dataset on website

    midto late-October

    Second release of APS master dataset uploaded to website

    Timeline


  • Login