Challenges and Evolution of Telephone Survey Sampling Methods

1. GfK NOP Social Research GfK. Growth from Knowledge RDD Sampling for Telephone Surveys Nick Moon, GfK NOP Social Research

2. 1 • Early History of RDD 2 • First successful method Agenda 3 • Becoming mainstream 4 • A new challenge 5 • Future challenges

3. The Theoretical Case • Telephone generally cheaper than face to face • Unclustered no dearer than clustered • Possible advantages for sensitive questions • More questions per minute • Little evidence of mode effects • Shortage of face to face fieldwork

4. Telephone Interviewing • Took off from early 1980s • Rapidly increased proportion of total research volume • But very rarely used for social research • Initially concerns about penetration • penetration >95% • Concerns also about sampling • Need for probability sampling

5. Probability Sampling • Needs a sample frame with 100% coverage • Only frame for telephone is the telephone directories • Considerably <100% coverage • High and growing ex-directory rate • UK 36% overall London 50%+ (1999)

6. Ex-Directory Subscribers • Urban • Younger • Female • Lowest social class groups • Higher income • Smaller households

7. A theoretical alternative • While addresses or name lists are infinite, the number of telephone numbers is finite • It should therefore be possible to generate numbers entirely at random from the known list of possible numbers

8. But not so good in practice • A ten digit telephone numbering system allows 1 billion numbers • only 25 million households in the UK • many numbers are business only and not residential • vast majority of possible numbers are not in use • The cost of finding live numbers can be prohibitive • Foreman and Collins (1991) • Lack of information about the numbering system

9. The first compromise • RDD is not cost-effective • Directory sampling is biased • Directory plus 1 in theory compensates for this bias • Later developed into directory plus n

10. Problems of Directory plus n • Unlisted numbers tend to be clustered • If 10 consecutive numbers are unlisted then directory + any number from the first one will yield no listed number • the chance of any number being selected depends on the proportion of the previous nine numbers that are in the directory

11. Number Propagation • Developed by BMRB in 1991 • Collect telephone numbers from all respondents on their Omnibus • This includes ex-directory numbers (though some will still refuse) • Generate numbers from n-3 through to n+3 • Has advantages over directories, but still effectively plus n rather than RDD • BMRB now use RDD

12. Telephone Interviewing • The 01727 exchange allows 1 million numbers, for a town of 50,000 people • But all start with either 8 or 7 • This immediately reduces total possible to 200,000 • More information now available from Ofcom over and above exchange listings • Can identify blocks of 100,000 numbers that you know exist • Working blocks still contain huge numbers of non-allocated numbers that must be winnowed out

13. 2 The First Success

14. The US example • Random Digit Dialling (RDD) used widely in the US • Initially Mitofsky-Waksberg • Subsequently list assisted

15. Mitofsky Waksberg 1 • All US numbers are in format 123-456-7890 • All possible combinations 123-456 known • Draw random sample of area code+local exchange+ random number from 00 to 99 as primary sampling units • The “100 Block” • Generate at random a number from 00 to 99 to produce one full number per psu • Telephone each of these numbers

16. Mitofsky Waksberg 2 • If that number rings, assume the block is in use, and keep in sample • If that number doesn’t ring, assume the block is not in use, and reject from sample • Because the chance of one selected number ringing is higher the more working numbers there are, then this is effectively pps sampling

17. Mitofsky Waksberg 3 • Since psu’s are selected pps, there should be a constant number of sample members per psu • Generate more random numbers from 00 to 99 to produce more numbers within the same 100 Block • Ring these numbers until the prescribed number of WORKING numbers has been reached • These are then treated in the same way as any other random sample with cal-backs, no replacement etc

18. A quick quiz Paper title: “Forty eight red, white and blue shoestrings” Why?

19. Forty eight red, white and blue shoe strings 'Mac the Finger said to Louis the King, I got forty eight red, white and blue shoe strings, And a thousand telephones that will not ring, Tell me where I can rid of these things' Bob Dylan, Highway 61 Revisited

20. The Moon–Noble experiment • Translation of Mitofsky Waksberg to UK hampered by lack of transparency about UK numbering system • BT cited concerns about privacy • Also irregular number length (9 digit, 10 digit) • Empirical mapping of number system using large scale tele-marketing databases • 15 million numbers, so can map nearly the whole system •  Identifies nearly all 100 banks currently in use •  Random sample of 100 banks for Mitofsky Waksberg approach •  Yielded 66% working lines at second stage

21. RDD v Face to face 1

22. RDD v Face to face 2

23. 1997 General Election

24. Moon Noble Experiment conclusion • Clearly a success • Showed true probability telephone sampling could be done cost-effectively in the UK • Won Best Technical Paper award at 1998 MRS Conference • Sat back and waited for glory

25. 3 Becoming Mainstream

26. A Very Short-lived Glory • Growth of competition in the telephone supply market • Increased role for Oftel (now Ofcom) • Far greater transparency about the number system • And standardisation of length • Now possible for US techniques to be applied to the UK • Commercial suppliers now make RDD samples available to all • Specialist agencies now sell RDD samples • Epsem • List-assisted • Pre-dialled with auto-dialler to weed out non-working numbers

27. Two Different Approaches • True epsem • Does not take number of directoried lines into account • List Assisted • Glorified version of directory plus n • Directory minus n digits with last n digits randomised • List assisted will always be biased against blocks with large numbers of working numbers of which very few are directoried • This may or may not matter • Epsem will always produce far more non-working numbers • Use of pre-dialing or “pinging” to remove non-working lines

28. Issues of Geography • Less than perfect match between telephone numbers and geography • Fine at exchange code level • Progressively less good at more detailed levels • No official record of geography of numbers • Postcodes no longer printed in telephone directories • Commercial suppliers will sell samples based on various geographical levels • Assign numbers to postcodes based on available databases • Reasonable at constituency/local authority level • Poor at ward level • Need to oversample for ineligibles and allow for screening • People don’t always know where they live

29. Implications of Geographical Issues • No real problem for national surveys • Doesn’t really matter if isolated case is in the “wrong” psu • Considerable implications for local studies • Lose some of the cost advantages of telephone interviewing • Especially problematic for clustered samples • Such as locating areas of high density of BME populations

30. Evidence from the Staffordshire SE experiment • Allocation of numbers not geographically rational • The vast majority of numbers were in the format EEEEE-Lxxxxx • Where EEEEE was the exchange and all started with the same sub-exchange number L • A small number were in the format EEEEE-Nxxxxx • Presumably the L sub areas was filling up and a new sub-area N came into use • One might expect these new numbers to run sequentially from EEEEE-N00000, EEEEE-N00001, EEEEEN00002 etc • In fact they were scattered over huge range of numbers

31. Implications of this number scatter • The core of any RDD sampling approach is blocks of 100 or 1000 numbers • We have already examined the implications of different densities of directoried numbers per block • Relative density of working numbers per block is also important • Scatter of the Staffordshire kind will lead to a density distribution with a hugely long tail • In theory pps sampling should take care of this • A small proportion of very low-density blocks will still get selected • But most RDD systems rely on finding a fixed, and non-trivial, number of working numbers per block • Nicolaas (2001) suggests very low density blocks might be safely ignored

32. Sampling Individuals • A telephone is usually a household item • But we want a sample of individuals • Quota approach can take whoever answers the phone • Old social rules on who answers phone have disappeared • New ones appeared in their place • Especially in households with teenagers • Random selection is ideal

33. Random Selection of Individuals • Kish Grid is the gold standard • Enumerate household in set order • Take nth person list according to randomised procedure • Requires enumeration at very beginning of interview • Felt to cause potentially higher refusal rates • NatCen experiment suggests this may not be so • Next/last birthday method a simpler compromise • NatCen experiment shows similar profile to Kish grid • The Rizzo/Brick/Park variant even easier • How many adults in household? • If one continue interview • If two randomly select either the person who answered the phone or the other one • If three or more go through next birthday method

34. Weighting the data • People in larger households have less of a chance of selection than those in smaller ones • Same principle as face to face surveys using PAF • Effective sample size only c80% of actual one • Households with >1 line have >1 chance of selection • Need question on number of lines • But if the line is used only for a modem/fax then it shouldn’t count • Need question on lines that ring • All adds to the cost

35. A Cautionary Note • There’s more to random sampling than just sampling • Many agencies buy RDD samples but treat them as leads for a quota sample • If you get a response rate of 15% does it matter if the sample is random?

36. 4 A New Challenge

37. 2003 Communications Act • Gives Ofcom the power to deal with silent calls • Primarily directed at tele-marketers • Unexpected impact on RDD

38. Silent calls • Caused by use of auto-diallers • Machine dials number • When it is answered machine hands it over to any available interviewer/salesman • If all are busy respondent gets “silent call” • Not cut off but no-one there

39. Impact of Silent Calls • Silent call rate can be set to any level on dialler • Above that rate and system stops making calls • The higher the rate the less interviewer salesman dead time • But more pissed-off recipients of silent calls • Lots of tele-marketers don’t care, MR firms generally do • GfK NOP silent call rate always 1% or under

40. Consequences for RDD • The only supplier of epsem samples temprarily ceased to do so • They were concerned pinging may be against the new law • Without pinging epsem is not cost-effective • Now supplying pinged sample again - BUT • The main clients for true random samples are government bodies • Concern about seeming to be against spirit of legislation

41. 5 Future Challenges

42. Mobile Phones • Only 1% of households have no phones • But 8% have only mobiles and no landline • These are being missed from RDD surveys • Does it matter?

43. Problems of Mobile Phones • Mobiles are individual • Landlines are household • Mobiles may be provided by employer • Mobiles may be answered while driving • NB in the US many cell phone users pay to receive calls

44. Mobile-only households • Younger • 38% of those with a mobile but no landline are 18-24 • Down-market • 58% of those with a mobile but no landline are social class DE • More likely to be in full time education • More likely to be unemployed

45. Dealing with Mobiles • Mobile-only households will reach a level where they can’t be ignored • Mobile numbers identifiable • Could sample in the same way as landlines for RDD • Multiple chances of selection • Need to ask for number of landlines and number of mobiles • No geographical data

46. The Future’s Orange/Virgin/O2/Vodaphone • Further Changes to the Numbering System • Particularly influenced by competition among suppliers • Ofcom keen to reduce barriers to competition • Number Portability • Must be same broad area BUT • VOIP • Mobile/landline hybrids • Wireless Office

47. Implications of future changes • The link between telephone number and geography may break down completely • National samples can still be drawn by simple random sampling, but with no benefits of stratification • Local sampling by telephone may become impossible • Cost may force local surveys onto postal or online methodologies

48. The Paradox • The demand for high quality high volume social research is beginning to exceed the supply of suitable face to face fieldwork • This is diverting social research to telephone, that previously would only ever have been conducted face to face • This research demands high quality epsem samples, at the very time that it is being more difficult to achieve them • What’s wrong with paper self-completion anyway?