AN EXPERIMENT WITH CATI IN ISRAEL

Transcription

1 Paper presented at InterCasic 96 Conference, San Antonio, TX, Background AN EXPERIMENT WITH CATI IN ISRAEL Gad Nathan and Nilufar Aframian Hebrew University of Jerusalem and Israel Central Bureau of Statistics In recent years, telephone interviewing has become the predominant mode of data collection for household sample surveys in North America and in Western Europe. The U.S. Subcommittee on Computer-Assisted Survey Information Collection (1990) concluded that CATI has become a standard collection vehicle grounded strongly in a firm body of research. The use of the telephone for surveys of households and individuals has only recently become common in Israel, primarily by commercial survey organizations. The late entry into the area of telephone surveys in Israel has been due primarily to the relatively slow development of the telephone system until several years ago. Thus in 1983 only 69% of households in Israel had telephones (Central Bureau of Statistics, 1985). By the end of 1992 this rose to an estimated 93% (Central Bureau of Statistics, 1994). However there still remained large differentials in telephone ownership, e.g., from 71% for the lowest decile of household net income to 99.5% for the highest. Based on an estimated increase in direct telephone lines of 29% between 1992 and 1995 (not including cellular phones), educated guesses put present home telephone coverage in Israel as close to saturation. Israeli commercial survey organizations have in recent years increasingly used the telephone, in part computer assisted, for opinion polling and market research. However very little information is available on the methods they employ and there are doubts about the quality of their work, especially as reflected by the results of political polls and their relationship (or lack of it) to election outcomes. On the other hand, the use of the telephone for surveys of households and individuals by the research community and by the public sector has been rather sporadic and very little of it is computer assisted. While telephone interviewing is widely used by the Central Bureau of Statistics for surveys and for data collection relating to establishments (recently with computer assistance), no ne of the Bureau s household surveys are completely based on telephone interviewing. In the past, at least, this was due primarily to the low degree of coverage, but doubts about quality and methodological limitations have also played a role. However, telephone interviewing has been used partially in the quarterly Labour Force Surveys for over fifteen years. A study, by Nathan and Eliav (1988), compared the quality of telephone interviewing with that of home interviews in the Labour Force Survey, based on a misclassification model. The results gave some indication that response errors can be reduced by using telephone interviewing. The first academically based CATI facility in Israel was set up at the Hebrew University in 1994 to help researchers develop telephone surveys on substantive issues using modern methods and to carry out methodological research on the use of computer assisted telephone interviewing in Israel. The facility presently comprises eight stations, each

2 consisting of a 486 PC, connected to a local network, and a telephone line with a commercial dialer. The software used is CASES 3.2 of the University of California at Berkeley's CSM, in conjunction with an interface to a dialing software developed locally. Both commercial market research organizations and academic units, who base their survey work on the telephone, use some version of the telephone directory as their framework (except for the surveying of special sub-populations for which lists are available). However practically nothing is known about the coverage or accuracy of the phone company s directories. In fact after repeated attempts to obtain even a rough estimate of the proportion of unlisted numbers, the telephone company admitted that they could not obtain an estimate, due to technical difficulties. The methodological test reported on in the following was initially designed to obtain information for advising the Television Authority on the statistical aspects of evaluation methods for TV rating measurements. The use of telephone interviewing had been advocated but there was no information available on the characteristics of alternative sampling strategies, under local conditions. The possibilities considered were.(1) sampling from commercial versions of the telephone directory or (2) Random Digit Dialing (RDD) - using the well-known Mitofsky-Waksberg method (Waksberg, 1978). 2. Methodological test design The methodological test was primarily designed to test the use of two alternative sampling designs and to assess and compare the effects of undercoverage (unlisted numbers), out-of-scope cases (e.g. non-households), non-contacts, refusals and differential response error effects of the two sample designs. The test was limited to the municipal area of Jerusalem. Some six hundred telephone numbers were selected by each of the methods and telephone interviews were carried out on television ownership and use. Five students, without previous interviewing experience, recruited especially for the task, carried out interviews. They were given brief written and oral instructions and carried out a short practice training run. Interviews were made, primarily during the evening hours, at the university CATI laboratory, in the presence of a supervisor, who provided the sample lists of phone numbers and helped out in cases of difficulties. Interviewers received telephone numbers from both frameworks, without knowing the source, and the interviewing instruments were identical for both samples. The sample from the telephone directory was obtained from a commercial organization, which extracts the phone numbers from the electronic version of the published phone directory. The differentiation between home phone numbers and commercial numbers is based on algorithms, which analyze the text of the registered name. An initial simple random sample of 1000 numbers purported to relate to home phones in the Jerusalem dialing area was obtained. Of these 274 turned out to be in localities outside the municipal borders of Jerusalem and a simple random sub-sample of 546 was selected from the remaining 726 numbers. 2

3 The RDD sample was based on a listing provided by the phone company of 217 prefixes. These were of three digits for six digit numbers and of four digits for seven digit numbers and covered all exchanges within the municipal area with at least one home phone. Thus non-working exchanges and exchanges with only commercial or institutional phones were excluded. A simple random sample, with replacement, of 33 prefixes was selected. The Mitofsky-Waksberg sampling scheme with target sample size of 10 responding households per sampled prefix was applied. Thus prefixes for which the first number dialed was not a household (commercial, institution or not connected) were not sampled further. Prefixes for which the first sampled number represented a household, were further sampled (without replacement) until 10 responding households were obtained. In practice, strict adherence to the design was not obtained because of the high rate of nonresponse. Thus in some cases there were slight differences in the final cluster sizes for valid prefixes. Out of the 33 sampled prefixes, three were excluded, since the first sampled number did not belong to a household. A further four were considered as nonvalid, since no household was reached in after 3-5 attempts for each of a sample of twenty numbers. The remaining 26 valid prefixes yielded a sample of 274 identified households (i.e. an average of 11.0 households per valid prefix), of which 193 were interviewed - see detailed results below. Thus the final average yield per valid prefix was 7.4 interviewed households. Interviewing was carried out during the months of June through August between 7 and 9 PM. Non-contacts - no answer, busy signals, answering and fax machines or cases of response where it could not be determined whether the number was a household or not - were called at least three and up to five times to attempt to determine their status. Attempts were made to call non-contacts at special sessions during the afternoon (4 to 7 PM). Once it was determined that the number did relate to a household, additional calls were made in case of non-response due to inconvenient timing or language difficulties. Supplementary interviews were carried out by Russian and Arabic speaking interviewers. Additional attempts were made to call sub-samples of non-response three more times to get some indication of their status. For households where an interview could be obtained, questions were asked about the number of TV sets in the household, whether connected to cable TV, programme watched at the time of the interview (for each TV set) and number of persons watching each set. In addition the number of different telephone line connections used for voice calls was asked (for correct weighting). Finally, one individual was selected at random in each household (by the last birthday method) and, if available for interview, was asked about watching news programmes the previous evening. Up to three call-backs were made to interview individuals who were not available at the time of first interview. In addition, a small sample of phone numbers was selected from among those defined by the supplier of the directory framework as non-households (businesses or institutions) and called to check their status. Also the sample of RDD households was matched to the commercial framework of households to assess the undercoverage of the directory due to unlisted numbers. 3

4 3. Results 3.1 Sample administration The following table summarizes the administration results of the two samples. The percentage results for the RDD sample are unbiased estimates, obtained as weighted averages (weighted by final cluster size, which ranged from 1 to 12), whereas the absolute numbers are unweighted. Since the directory sample was a simple random sample, no weighting was carried out for its results. Weighting to take into account households with more than a single telephone line (see below) was not carried out at this stage, since the information was only available from respondents. Samplin g errors for percentages were estimated taking into account the implemented sample design. The results include those obtained in the supplementary calls made to N/A s in the initial calls. A high proportion of call attempts (30-32% - similar for both samples) were unsuccessful even in obtaining the information necessary in order to ascertain whether the telephone number represents an in-scope household or not, even after 3-5 callbacks. It should be noted that automatic changes in telephone numbers (i.e. tho se announced by a voice message) were immediately followed up at the new number, for both samples. The experiment was carried out during the transition of some numbers from six to seven digits by the addition of a first digit and this change could be made automatically for the both samples before dialing. Non-automatic number changes, requiring directory assistance, were treated differently for the two samples. For the directory sample, an attempt was made to follow-up on these changes (under the assumption that they were not represented elsewhere in the directory framework). However this was not done for the RDD sample, since the changed number (if in the experiment area) was represented in the RDD framework. Thus number changes, for which the new number could not be ascertained, were considered as unknown for the directory sample and as out-of-scope for the RDD sample. 4

5 Table 1: Administration results (sample errors in parentheses) Directory Sample Num. Pct. Num. Pct Num Pct Total sample N/A whether in-scope (2.0) of which: No reply (3.7) Answering machine (2.5) Engaged or wait-call (3.1) Fax, modem, number changed, (2.3) Outcome known - total (2.0) of which: Out-of-scope (1.7) Business/institution Inactive number (7.3) In-scope (1.7) Interviewed Not interviewed (2.7) 5

6 Table 1: Administration results (sample errors in parentheses) (cont.) RDD Sample Num. Pct. Num. Pct. Num. Pct. Total sample N/A whether in-scope (4.4) of which: No reply (5.6) Answering machine (2.8) Engaged or wait-call (3.8) Fax, modem (4.5) Outcome known - total (4.3) of which: Out-of-scope (6.5) Business/institution Inactive or number changed In-scope (6.5) (6.5) Interviewed Not interviewed (5.1) The distribution of reasons for N/A results of call attempts differs somewhat between the two samples. Thus the RDD sample N/A s had a higher proportion of no reply (73%) than the directory sample (61%), while the latter had higher proportions of engaged numbers and of answering machines. This probably reflects a higher proportion of out-ofscope numbers (businesses and institutions) in the RDD sample. There is obviously a large difference between the outcome distributions of the two samples, for those calls where it could be ascertained if they represented in-scope households or not Thus the RDD sample yielded 40% out-of-scope numbers, whereas only 12.6% of the ascertained directory numbers were out-of-scope. However, the latter 6

7 number is also rather high for a framework which is claimed by its producers to contain only valid household numbers. The detailed breakdowns also differ - almost half the directory out-of-scopes are non-households, as against only a third of the RDD cases. For in-scope households there is a significant difference in completion rates between the two samples. Less than 60% of the in-scope directory sample households were interviewed, as against 75% of the in-scope RDD sample households. This result was unexpected, since the RDD sample includes unlisted numbers not included in the directory framework, who would a-priori be considered less likely to agree to be interviewed. Also, a slightly higher proportion of non-response was due to refusals (not shown in the table) for the directory sample (75%) than for the RDD sample (68%) - but the difference is within sampling error. The remaining non-response was due primarily to language problems - 14% and 24% respectively (notwithstanding the attempts to use Russian and Arabic). The residual non-response - due to lack of an adult respondent or inconvenient timing - was 10% and 8%, respectively. The difference in response rates between the directory sample and the RDD sample is in part due to differences in the eligibility rules. A basic standard definition of a survey response rate, which takes this difference into account, has been proposed by the Council of American Survey Research Organizations - Frankel et al. (1982). They recommend that units of unknown status be allocated to eligible and ineligible categories in the same proportion as among the units of known status. Applying this definition to the results in table 1, standardized response rates of 40.4% for the directory sample and of 47.7% for the RDD sample are obtained. 3.2 Substantive comparisons Comparisons between the main results obtained from the directory sample and those obtained from the RDD sample are given in the following table. The results are weighted to ensure unbiased estimation, based on the sample design. Thus the RDD results are weighted inversely by the cluster size and in both samples, households with multiple phone lines are weighted by the reciprocal of the number of lines and individuals weighed by the number of adults in the household. 7

8 Table 2: Substantive results (weighted) Directory sample RDD sample Households Percent S.E. Percent S.E. Number of telephone lines: Number of TV sets: Cable TV * : Yes No Channel viewed at time of interview * : None Channel Channel Cable channel Individuals Percent S.E. Percent S.E. Last night news viewing * : None Channel Channel Cable Education (years of study):

9 The results show practically no difference in the number of phone lines between the two samples - about 30% in both samples being connected to more than one line. Note that this includes only fixed lines - cellular phones are not covered by either framework. Although the use of cellular phones has increased very rapidly lately, it still seems extremely rare for households to give up the use of a fixed line. The number of TV sets reported by the RDD sample seems to be substantially lower than that reported by the directory sample. The number with no set at all is very high (42%) for the RDD sample as compared to 26% for the directory sample, though the difference is not significant (at the 5% level). The proportion owning more than one set is significantly higher for the directory sample (30%) than for the RDD sample (18%). Also among those owning TV sets, the number with access to cable TV is lower for the RDD sample (46%) than for the directory sample (61%), though the difference is not significant. Similarly, TV viewing (at the time of the interview) is somewhat lower among those selected by RDD. Most of the difference is accounted for by the lower rate of viewing cable channels, so that this result is, at least in part, related to the previous one. On the other hand, individuals viewing of last night s news was somewhat higher for the RDD sample. Overall, it seems that the RDD sample has a pattern of characteristics with respect to TV, which is usually associated with higher socioeconomic levels. It is conjectured that this is due to the fact that the directory sample does not represent unlisted numbers (which are represented in the RDD sample) and that unlisted numbers are in general associated with higher socio-economic levels. This is further emphasized by the comparison of levels of education for the two samples, which shows a significantly higher level for the RDD sample. 4. Supplementary investigations 4.1 Non-response Samples of non-respondents in the initial stage (i.e. from those for whom it could not be ascertained whether they were in-scope or not) were selected for each sample - 67 (out of the 173) initial N/A s in the directory sample and 75 (out of the 202) initial N/A s in the RDD sample. The results are summarized in the following table: 9

10 Table 3: Supplementary sample results (from initial N/A s) Directory sample RDD sample Number Percent Number Percent Total sample of initial N/A s N/A whether in-scope Outcome known - total of which: Out-of-scope : In-scope The supplementary samples provide only limited information about the N/A s, since high proportions persisted in not providing even the information on whether they are in-scope - 28% for the directory sample and 48% for the RDD sample. However, of those whose status could be ascertained, the out-of-scope proportions were very similar to those obtained in the original sample (see Table 1) % and 12.6%, respectively, for the directory sample and 43.6% and 40.0%, respectively, for the RDD sample. This implies that the biases due to N/A s with respect to the out-of-scopes may be limited. 4.2 Unlisted telephones It is well known that the extent of unlisted telephones is a major factor in determining the bias due to the use of a directory framework, rather than RDD sampling (Waksberg, 1978). The extent of unlisted phones varies quite considerably from some 12% in Britain to over 20% in the U.S. (Collins and Sykes, 1987) and is estimated as 14% in France (Fréjean, Panzani and Tassi, 1990). As mentioned above no estimate of the extent of unlisted phones in Israel could be provided by the phone company, nor from any other source. The substantive results reported above do provide strong indirect evidence for substantial biases in the use of the directory as a sole sample framework for telephone surveys. An attempt was made to assess the extent of unlisted phones or other undercoverage of the directory framework. A list of 231 telephone numbers from the RDD sample, for which it could be ascertained that they corresponded to a household, was given to the commercial company who provided the directory listing. They were requested to match the list provided with their own listing. Only 168 were found in their listing. This leads to the very high estimate of some 27% undercoverage in the directory listing. While most of this would most probably be due to unlisted numbers, some of it may be due to lacunae in the process used by the company in obtaining their lists. It should be noted that the match was against the company s complete listing (for the 10

11 Jerusalem dialing area) and not only against those they considered as households. It was conjectured that part of the unlisted phones were of second phone lines in multi-line households. However among those for whom the information was obtained a far higher proportion of multi-line households was found among those included in the directory listing (24%) than among those not listed (5%). In any case the results do indicate that the use of the directory listing may result in serious undercoverage. 5. Summary and conclusions This small scale experiment obviously suffers from very high non-response rates, both at the stage of ascertaining in-scope status and to a somewhat lower extent at the interviewing stage. Obviously much more effort would have to be put into reduction of N/A rates for real survey operations. As the supplementary investigations show this is possible at not too high additional costs, especially if call attempts can be spread over different times. For instance, one commercial organization reported that it obtained high response rates by calling during the early afternoon hours of Friday (before the start of the Sabbath). The total average time required per completed interview ranged from three to six minutes and was practically the same for both samples. Even if increased callback rates were to double these times, the reduction in non-response might make this a worthwhile investment. Although it is difficult to assess the effect of non-response on the comparison between the samples from the two sources, there is no doubt that there are important differences between the RDD and the directory samples in the final results. While we cannot be absolutely sure that this is due to unlisted numbers, it is very difficult to conceive of any other reason for the large differences found. The fact that all the differences manifest themselves in factors related to socio-economic characteristics (e.g., higher educational level, fewer TV sets, less cable TV and higher rates of news watching for the RDD sample) is consistent with the conjecture that the non-representation of unlisted numbers in the directory sample may lead to serious biases in any variable associated with socioeconomic level. These findings will certainly have to be verified in further systematic studies, carried out on a large scale. However, even at this point it seems obvious that the continued uncritical use of samples for investigating the general population, which are based solely on directory frameworks can no longer be justified. 6. References Central Bureau of Statistics (1985). Housing conditions and possession of household equipment Census of Population and Housing Publications, No. 9, Jerusalem. Central Bureau of Statistics (1994). Family Expenditure Survey 1992/93. Part B: General Summary. Special Series No. 975, Jerusalem. 11

12 Collins, M. and Sykes, W. (1987). The problems of non-coverage and unlisted numbers in telephone surveys in Britain. Journal of the Royal Statistical Society A 150, Frankel, L. R. et al. (1982). On the definition of response rates. Special Report of the CASRO Task Force on Completion Rates. New York: Council of American Survey Research Organizations. Fréjean, M., Panzani, J.P. and Tassi, P. (1990). Households with unlisted telephone numbers and telephone surveys. Journal de Societé de Statistique de Paris 131, Subcommittee on Computer-Assisted Survey Information Collection (1990). Computer- Assisted Survey Information Collection. Washington, DC : Office of Management and Budget Waksberg, J. (1978). Sampling methods for Random Digit Dialing. Journal of the American Statistical Association 73,