Conceptual and Operational Issues in the Measurement of Internet Use* Jonathan Zhu City University of Hong Kong firstname.lastname@example.org * Funded by the UGC of HKSAR (CityU1152/00H) @
Background: the Diffusion of the Internet in Hong Kong, Beijing and Guangzhou Source: J. H. Zhu (2003)
Issues in Measurement of Internet Use and Users • The size of “Internet users” in a society is a function of: • Definition of study population (SP) • Method of sample weighting (SW) • Requirement of minimal usage (MU) • The amount of “online time” by Internet users is a function of: • Definition of study population (SP) • Method of sampling weighting (SW) • Method of data collection (DC) • Treatment of extreme values (EV)
Criteria for Evaluation of Measurement • Validity: how accurate or correct is the measure as compared with the “truth”? • Reliability: how precise or stable is the measure over time and/or across space? • Practicality: how efficient or economic is the measure in data collection and analysis?
Data • Hong Kong Survey 2002: telephone interviews of 1,800 residents at 6 and above in Dec. 2002 by Jonathan Zhu and his team • AC Nielsen/Netratings 2002-03: online tracking of 1,500 Internet users from 811 households in Hong Kong in Oct. 2002 and Jan. 2003.
Definitions of Study Population • WIP-Hong Kong: 18-74 • CNNIC: 6+ • Another popular definition: 18+ • HK Census 2002: • 6-17: 16.4% • 18-74: 80.0% • 75+: 3.6%
Impact of Population Definitions on Internet User Size Data: Hong Kong 2002
Impact of Minimal Requirements on Internet User Size Data: Hong Kong 2002
Age Distribution of the Sample before and after Weighting Data: Hong Kong 2002
Impact of Weighting Methods on Internet User Size Data: Hong Kong 2002
Summary: Internet Users by Population, Usage Requirement & Weighting Method Data: Hong Kong 2002
A Mathematical Model of “True” Internet Users (TIU) TIU = 55.3 – 1.4SP18-74 - 3.7SP18+ - 4.5MU – 5.4SW (Adjusted R2 = 99.6%, Standard Error = 0.3%) Where TIU is the “Unadjusted” Internet Users (%) for HK in 2002, which should be 1.4% less for a study population of 18-74, or 3.7% less for a study population of 18+, or 4.5% less if those use the Internet less than 1 hour per week are excluded, or 5.4% less if the sample is weighted based on population census.
Impact of Population Definitions on Online Time (at Home) Data: Hong Kong 2002
Impact of Weighting Methods on Online Time (at Home) Data: Hong Kong 2002
Impact of Extreme Values on Online Time (at Home) Data: Hong Kong 2002
Impact of Data Collection (DC) Methods on Online Time Data: HKS 2002 & Netratings 2002-03
Summary: Online Time by SP, SW, DC, and EV Data: Hong Kong 2002
A Mathematical Model of “True” Online Time (TOT) TOT = 532 + 16SP18-74 – 22SW – 49EV - 249DC (Adjusted R2 = 93.5%, Standard Error = 34.3) Where TOT is the “Unadjusted” Online Time (min.) for HK users in 2002, which should be 16 min. more for a study population of 18-74, 22 min. less if the user sample is weighted, 49 min. less if extreme values are removed, or 249 min. less if data are collected through online tracking method.
Telephone interview data include: Online time at both home (68%) and elsewhere (32%); Non-HTTP based activities such as using POP3 Email (=136 min./week) and other protocols; Online tracking data include: Online time only at home; Only HTTP=based activities protocols). Caution: Different Definitions of “Online” Activities It is estimated that tracking data may measure only 51% of the total online time..
Estimated Distribution of Online Time by Location and Protocol of Usage
Conclusion: How Many Internet Users Are There? • The size of “Internet Users” is significantly affected by the definition of study population (SP), the requirement of minimal usage (MU) and the method of sample weighting (SW). • SP (e.g., general population vs. adults) may produce a difference of 1-4% and MU (e.g., no requirement vs. 1 hour per week) up to 5%. While there is no “correct” definition of SP or MU, it is important to report the definition and adopt, whenever possible, multiple definitions. • SW (weighted vs. unweighted) may contribute another 5% difference. Since Internet use is highly correlated with age and sex, it seems both necessary and effective to weight the sample to ensure the accuracy of the measurement.
Conclusion: How Much Time Do They Spend Online? • The amount of online time is marginally affected by SP (p = 0.3) and SW (p = 0.2) probably due to the fact the base of analysis is already restricted to users. • Online time is significantly affected by the treatment of extreme values (EV), which may inflate online time by up to 10%. It is thus necessary to control for it (i.e., removing EVs). • Online time is most significantly affected by the method of data collection (DC, e.g., interviews vs. online tracking), which may result in a difference of 2-folder. Although online tracking is generally more accurate, it is far more expensive and impractical in many societies. It is thus important to keep in mind the magnitude of inflation in self-reported data.
Ultimate Criteria for Evaluation • Validity: how accurate or correct is the measure as compared with the “truth”? • Reliability: how precise or stable is the measure over time and/or across space? • Practicality: how efficient or economic is the measure in data collection and analysis?
Consistency in Measurement of Internet Users over Time and across Space* * Based onWIP definition.
Final Verdicts • Measurement of Internet users and online time based on interviews data is largely reliable over time and across space. • The interview-based measurement is generally more practical than online tracking method. • The interview-based measurement is generally weaker in validity, as compared to online tracking method. However, it could be adjusted if the departure from the “truth” is known (e.g., based on comparison with online tracking data.