Impact of September 11, 2001 Events on Substance Abuse and Mental Health in the New York Area

Appendix C

Statistical Methods and Limitations of the Data

C.1 Target Population

An important limitation of NHSDA estimates of drug use prevalence is that they are designed to describe only the target population of the survey (e.g., civilian, noninstitutionalized persons aged 12 or older). Although this population includes almost 98 percent of the total U.S. population aged 12 or older, it does exclude some important and unique subpopulations who may have very different drug-using patterns. The survey excludes active military personnel, who have been shown to have significantly lower rates of illicit drug use (Bray et al., 1999). Persons living in institutional group quarters, such as prisons and residential drug treatment centers, are not included in the NHSDA and have been shown in other surveys to have higher rates of illicit drug use (Bray & Marsden, 1999). Also excluded are homeless persons not living in a shelter on the survey date, another population shown to have higher than average rates of illicit drug use.

C.2 Hypothesis Testing

This report is concerned with the effects of the September 11 terrorist attacks on substance use and other behavior in NYC and areas nearby.

Because the September 11 event occurred almost at the beginning of the fourth quarter of 2001, the impact of the event can be evaluated by comparing substance abuse and other measures of behaviors in the fourth quarter of 2001 with measures obtained in previous quarters.

A simple "pre-post" statistic is the difference between the level of a measure in quarter 4 of 2001 and its level in quarters 1 to 3 of the same year.

While the simple pre-post statistic is not unreasonable, observed differences may be due to seasonal variations in substance abuse, for example. To adjust for this seasonal variation, similar pre-post statistics were calculated for an aggregate of three unaffected urban areas. Definitions of the three geographic areas used in this report are given in Table C.1.

C.2.1 Analytic Approach

Our focus was on the potential effects of the September 11 events on behavior in the New York area in the fourth quarter of 2001. To be able to assess these effects, levels of drug use and other behaviors in that quarter, in NYC and in NYCMSA, were contrasted with levels in other time periods and/or with those levels in other areas.

Table C.2 shows the areas and time periods involved. The area denoted by NY can be either NYC or NYCMSA (see definitions in Table C.1).

In Table C.2, and in the definitions of the test statistics below, Q indicates an estimate of a level of drug use or other behavior. The fourth quarter of 2001 in NY is indicated in boldface, as this is where the effect of interest lies.

Thus, differences between Q_4,NY and the other time periods and areas may be used to make inference on the effects of interest.

We have assessed the September 11 effects by considering the following four contrasts:

Within-Area Contrasts. The first test is a simple within-area, within-year pre-post contrast. It is defined by

T₁ = Q_4,Area,2001 – Q_{1-3,Area,2001}

where Area = NYC, NYCMSA, or COMB.

The contrast T₁ may be deficient in confounding the September 11 and seasonality effects. Thus, we have also used a within-area contrast, adjusted for seasonality, by subtracting the estimated 2000 seasonality from T₁. This test statistic is given by

T₂ = [Q_4,Area,2001 – Q_{1-3,Area,2001}] – [Q_4,Area,2000 – Q_{1-3,Area,2000}].

When T₂ is applied to the NY areas, it compares the Q4 to Q1-3 differences in 2001 (consisting of both seasonality and September 11 effect) with the Q4 to Q1-3 differences in 2000 (consisting of seasonality alone).

The idea behind T₂ may be expressed in the following representation: Assuming (approximately) constant seasonality, this test statistic reflects the potential September 11 effect:

T₂ = [seasonality in 2001 + 9/11 effect] – [seasonality in 2000].

Thus, assuming seasonality did not change, T₂ estimates the September 11 effect.

Between-Area Contrasts. Pre-post within-year contrast, adjusted for seasonality by subtracting the estimated seasonality in the combined CMSA:

T₃ = [Q_{4,NYCMSA,2001} – Q_{1-3,NYCMSA,2001}] – [Q_4,COMB,2001 – Q_{1-3,COMB,2001}].

The idea behind T₃ is similar to that behind T₂:

T₃ = [seasonality in NYCMSA + 9/11 effect] - [seasonality in COMB].

Thus, T₃ represents the September 11 effect when the following assumptions are made:

• If there was a September 11 effect, it was only in NYCMSA, not in COMB.
• Seasonality in NYCMSA (apart from the September 11 effect), is the same in NYCMSA and in COMB.

Lastly, the seasonal effects may not be the same in the NYCMSA as they are in COMB. Further, the level of change (apart from a September 11 effect) from 2000 to 2001 may not be the same in these areas. The last statistic, T₄, does not need to make these assumptions. It is, however, more complex and involves all time periods crossed with geographic regions, as depicted in Table C.3.

The test is defined by

T₄ = {[Q_{4,NYCMSA,2001} – Q_{1-3,NYCMSA,2001}] – [Q_4,COMB,2001 – Q_{1-3,COMB,2001}]}
     – {[Q_{4,NYCMSA,2000} – Q_{1-3,NYCMSA,2000}] – [Q_4,COMB,2000 – Q_{1-3,COMB,2000}]}

C.2.2 Explanation

There are four terms in T₄. The difference between the first two terms consists of the potential September 11 effect and the difference in the seasonality between NYCMSA and COMB. The difference between the last two terms consists only of the difference in the seasonality between NYCMSA and COMB. Thus, the difference in the seasonality cancels out, leaving only the September 11 effect.

Assume the following simplified model for the level of substance use:

Q_{Quarter,Area,Year} = L_Year,Area + S_Quarter,Area + 9/11 effect (in NY) + survey year effect

where L_Year,Area is the level, and where S_Quarter,Area is the seasonal effect.

Q_{4,NYCMSA,2001} – Q_{1-3,NYCMSA,2001} = S_4,NYCMSA - S_1-3,NYCMSA + 9/11 effect

Q_{4,NYCMSA,2000} – Q_{1-3,NYCMSA,2000} = S_4,NYCMSA - S_1-3,NYCMSA

Q_4,COMB,2001 – Q_{1-3,COMB,2001} = S_4,COMB - S_1-3,COMB

Q_4,COMB,2000 – Q_{1-3,COMB,2000} = S_4,COMB - S_1-3,COMB

By substituting these in the formula defining T₄, we get the result that T₄ = September 11 effect. Note that, in this case, it was not assumed that the seasonality effects were the same in both regions; neither was it assumed that there was no survey year effect.

A few relationships exist between the four test statistics discussed above.

Test statistics T₁ and T₂ are related by

T₃ = T₁(2001) – T₁(2000).

Test statistics T₁ and T₃ are related by

T₃ = T₁(NYCMSA) – T₁(COMB).

Test statistics T₃ and T₄ are related by

T₄ = T(2001) – T₃(2000).

Finally, test statistics T₂ and T₄ are related by

T₄ = T₂(NYCMSA) – T₂(COMB).

C.3 Sampling Error and Statistical Significance

The sampling error of an estimate is the error caused by the selection of a sample instead of conducting a census of the population. Sampling error is reduced by selecting a large sample and by using efficient sample design and estimation strategies, such as stratification, optimal allocation, and ratio estimation.

With the use of probability sampling methods in the NHSDA, it is possible to develop estimates of sampling error from the survey data. These estimates have been calculated for all prevalence estimates presented in this report using a Taylor series linearization approach that takes into account the effects of the complex NHSDA design features. The sampling errors are used to identify unreliable estimates and to test for the statistical significance of differences between estimates.

C.3.1 Variance Estimation

Estimates of proportions, such as drug use prevalence rates, take the form of nonlinear statistics where the variances cannot be expressed in closed form. Variance estimation for nonlinear statistics is performed using a first-order Taylor series approximation in RTI's SUDAAN software package (Shah, Barnwell, & Bieler, 1996). The approximation is unbiased for sufficiently large samples and has proven to be at least as accurate and less costly to implement than its competitors, such as balanced repeated replication or jackknife methods (Rao & Wu, 1985).

C.3.2 Statistical Significance of Differences

This section describes the methods used to compare the prevalence estimates in this report. Customarily, the observed difference between estimates is evaluated in terms of its statistical significance. "Statistical significance" refers to the probability that a difference as large as that observed would occur due to random error in the estimates if there were no differences in the prevalence rates for the population groups being compared. The significance of observed differences in this report is reported at the 0.05 level. When making comparisons between, for example, the Q_{4, 2001} and Q_{1-3, 2001} estimates, one can test the null hypotheses (no difference in the Q_{4, 2001} and Q_{1-3, 2001} values) against the alternative hypothesis (there is a difference in these values) using the standard difference in proportions (or means) test expressed as

,

where p₁ = Q_{4, 2001} estimate, p₂ = Q_{1-3, 2001} estimate, var(p₁) = variance of Q_{4, 2001} estimate, var(p₂) = variance of Q_{1-3, 2001}, and cov(p₁,p₂) = covariance between p₁ and p₂.

Under the null hypothesis, Z is asymptotically distributed as a normal random variable. Calculated values of Z can therefore be referred to as the unit normal distribution to determine the corresponding probability level (i.e., p value). Estimates of Z along with its p value were calculated using RTI's SUDAAN, using the analysis weights and accounting for the sample design.

When making comparisons of estimates for different population subgroups from the same data year, the covariance term, which is usually small and positive, was ignored. This results in somewhat conservative tests of hypotheses that sometimes fail to establish statistical significance when in fact it exists.

C.3.3 Suppression Criteria for Unreliable Estimates

Estimates considered to be unreliable due to unacceptably large sampling error are not shown in this report and are noted by asterisks (*) in the tables and figures containing such estimates. The criterion used for suppressing estimates was based on the relative standard error (rse), which is defined as the ratio of the standard error (se) over the estimate.

Proportion estimates (p) within the range [0<p<1], rates, and corresponding estimated number of users were suppressed if:

rse[(-ln(p)] > 0.225 when p < 0.5
or
rse[(-ln(1-p)] > 0.225 when p > 0.5

Using a first-order Taylor series approximation to estimate rse[(-ln(p)] and rse[(-ln(1-p)], we have the following, which was used for computational purposes:

se(p)/p > 0.225 when p < 0.5
-1n( p)
or
se(p)/(1-p) > 0.225 when p > 0.5.
-1n(1-p)

The difference in formula for p < 0.5 versus p > 0.5 produces a symmetric suppression rule; that is, if p is suppressed, then so will 1- p. This is an ad hoc rule that requires an effective sample size in excess of 30. The effective sample size is defined as the sample size under a simple random sample design that will yield the same precision (Kish, 1965, p. 162). When 0.05 < p < 0.95, the symmetric properties of the rule produces a local maximum effective sample size of 42 at p = 0.5. Thus, estimates with these values of p along with effective sample sizes falling below 42 will be suppressed. A local minimum effective sample size of 30 occurs at p = 0.2 and again at p = 0.8 within this same interval, so estimates will be suppressed for these values of p with effective sample sizes below 30.

In other NHSDA surveys, this type of varying sample size restriction sometimes produced unusual occurrences of suppression for a particular combination of prevalence rates. For example, in other NHSDA surveys, lifetime prevalence rates near p = 0.5 were suppressed while not suppressing the corresponding past year or past month estimates near p = 0.2. To reduce the occurrence of this type of inconsistency, a minimum effective sample size was added to the suppression criteria. As p approaches 0.00 or 1.00 outside the interval (0.05, 0.95), the suppression criteria will still require increasingly larger effective sample sizes. For example, if p = 0.01 and 0.001, the effective sample size must exceed 92 and 414, respectively.

In addition, a minimum nominal sample size (n=60) is required to protect against unreliable estimates caused by small design effects and small nominal sample sizes. Prevalence estimates were also suppressed if they were close to zero or 100 percent (i.e., if p < .00005 or if p > .99995).

Estimates of continuous variables in this report (i.e., estimates of means) were suppressed if the relative standard error was greater than 0.5 or if the estimate was based on a nominal sample size less than 10.

C.4 Nonsampling Errors

Nonsampling errors can occur from nonresponse, coding errors, computer processing errors, errors in the sampling frame, reporting errors, and other errors not due to sampling. Nonsampling errors are reduced through data editing, statistical adjustments for nonresponse, close monitoring and periodic retraining of interviewers, and improvement in various quality control procedures.

Although nonsampling errors can often be much larger than sampling errors, measurement of most nonsampling errors is difficult or impossible. However, some indication of the effects of some types of nonsampling errors can be obtained through proxy measures, such as response rates, and from other research studies.

C.4.1 Response Rates

The nature of data collection for the NHSDA makes it possible to compare screening and interview response rates prior to and after the events of September 11. The bulk of all screening and interviewing field work is generally completed in the first two months of each quarter and the third month of each quarter is used to completed these activities. Because September 11 occurred in the third month of Quarter 3, 2001, national field staff had completed most of the screening and interviewing work for that quarter.

Table C.4 presents the weighted screening and interview response rates for NYC, the NY CMSA, and the C-CMSA by age group and gender. As shown, the interview and screening response rates generally decreased in NYC and NY CMSA from the first three quarters of 2001 to the fourth quarter, while rates were more stable in the combined CMSAs. In 2000, however, screening and response rates generally increased for most groups in all three areas between the first three quarters and the fourth quarter.

C.4.2 Inconsistent Responses and Item Nonresponse

Among survey participants, item response rates were above 98 percent for most questionnaire items. However, inconsistent responses for some items, including the drug use items, are common. Estimates of substance use from the NHSDA are based on the responses to multiple questions by respondents, so that the maximum amount of information is used in determining whether a respondent is classified as a drug user. Inconsistencies in responses are resolved through a logical editing process that involves some judgment on the part of survey analysts and is a potential source of nonsampling error. Because of the automatic routing through the computer-assisted interviewing (CAI) questionnaire (e.g., lifetime drug use questions that skip entire modules when answered "no"), there is less editing of this type than in the paper-and-pencil interviewing (PAPI) questionnaire used prior to 1999.

In addition, less logical editing is used because with the CAI data, statistical imputation is relied on more heavily to determine the final values of drug use variables in cases where there is the potential to use logical editing to make a determination. The combined amount of editing and imputation in the CAI data is still considerably less than the total amount used in prior PAPI surveys. For the 2000 CAI data, for example, 3.2 percent of the estimate of past month hallucinogen use is based on logically edited cases and 5.4 percent on imputed cases, for a combined amount of 8.6 percent. The combined amount of editing and imputation for the estimate of past month heroin use is 5.0 percent for the 2000 CAI.

C.4.3 Validity of Self-Reported Use

NHSDA estimates are based on self-reports of drug use, and their value depends on respondents' truthfulness and memory. Although many studies have generally established the validity of self-report data and the NHSDA procedures were designed to encourage honesty and recall, some degree of underreporting is assumed. No adjustment to NHSDA data is made to correct for this factor due to a number of studies addressing the validity of self-reported drug use data (e.g., Harrell, 1997; Harrison & Hughes, 1997; Rouse, Kozel, & Richards, 1985). The methodology used in the NHSDA has been shown to produce more valid results than other self-report methods (e.g., by telephone) (Aquilino, 1994; Turner, Lessler, & Gfroerer, 1992). However, comparisons of NHSDA data with data from surveys conducted in classrooms suggest that underreporting of drug use by youths in their homes may be substantial (Gfroerer, 1993; Gfroerer, Wright, & Kopstein, 1997; Bullington & Ginsberg, 2001).

C.4.4 NHSDA Field Interviewer Perspectives on How September 11 Influenced Respondent Behavior

In March 2002, NHSDA field interviewers gave their impressions during focus groups about how the September 11 terrorist attacks had influenced the behavior of survey respondents. Issues included changes in response rates, such as the number of calls required to make initial contact; the discontinuation and resumption of interviewing; and other changes in respondents' behaviors, including their interactions with the field interviewers. A total of 21 interviewers (19 women and two men) took part in four focus group sessions, all of which were conducted by telephone conference calls. Two of the groups were composed of field interviewers selected from across the United States but outside the New York and Washington, DC, metropolitan areas (11 interviewers working in nine states immediately before and after the attacks). Another focus group was composed of five interviewers working in Washington and southern Maryland, and the remaining group of five interviewers were working in NYC and nearby portions of Connecticut and New Jersey before and after the attacks. The following summaries of their comments are grouped according to three locations where the interviewers worked: metropolitan areas of New York and Washington, and a national group from across the nation but outside the New York and Washington areas.

New York. The New York metropolitan area interviewers generally agreed that some respondents were more willing to participate in NHSDA following September 11 but that this effect was temporary. The interviewers indicated that some respondents were more willing to participate for patriotic reasons, because of the survey's association with the government, and out of a general desire to help in the aftermath of the attacks. However, the interviewers also indicated that they faced difficulties resulting from terrorism. For example, more respondents indicated that they had not read NHSDA lead letters, which are mailed to potential respondents informing them about a future home visit by an interviewer. This behavior may have been influenced by the anthrax letters received in the New York area, Washington, and Florida during the months following September 11, which killed five people and infected many others.

Some of the other findings from the New York group of five interviewers are these:

• Regarding the lead letter, the discrepancy between the postmark and letterhead address made some recipients suspicious.
• One interviewer believed more people were at home following the attacks, one believed fewer were at home, and two saw no difference.
• In an area where people had lost their jobs as a result of the attacks, the number of refusals were relatively high.
• One interviewer used September 11 as a way to encourage respondents to participate, saying "government puts higher priority" on the survey following the attacks, and one interviewer used President Bush's State of the Union request that people contribute their time to community service as a way to encourage participation.

• People from the Middle East appeared to be nervous about government officials visiting their homes.
• Interviewers said bigger badges are better, adding that some respondents have commented that the lead letter and smaller badge looked unofficial or homemade.
• A few respondents did not wish to lick the verification envelope seal, due to the anthrax scare.
• Traffic and transportation problems for interviewers were worse in NYC following the attacks.

Washington, DC. The field interviewers in Washington, DC and southern Maryland reported that, following the attacks, their badges were more helpful in convincing people of their legitimacy. There were no respondent comments regarding anthrax. However, field interviewers believed the anthrax letters caused a delay in mail delivery, which prevented some respondents from receiving their NHSDA introductory lead letters prior to the home visits. The interviewers believed their ability to contact respondents in some households was somewhat reduced due to heightened security measures, particularly among respondents living in college residences. Also, some military and National Guard members could not be reached because they had been called to active duty as a result of the attacks and therefore were away from their homes. Interviewers said their travel was more difficult following the attacks due to additional security measures, including new parking restrictions and road closings.

Some other findings from the five interviewers from Washington and southern Maryland are these:

• Interviewers generally felt respondents were reassured by the lead letter, but the letters were less reassuring in inner-city neighborhoods.
• Because of tighter security at such places as college campuses, interviewers recommended that managers coordinate NHSDA work on campuses with campus administrators.
• Larger badges were "more helpful" in gaining respondent confidence.
• One of the five interviewers noticed significantly more refusals after the attacks, but another saw no difference in respondent cooperation. One interviewer noticed a difference in how respondents reacted but not in their willingness to participate.

National. The field interviewers from nine states1 outside of the Washington and New York metropolitan regions indicated that the events of September 11 influenced a few respondents to be more willing to participate but that this effect was temporary. An African-American female interviewer said that white males are typically the most reluctant group for her to interview but that white males seemed to take more time to listen following the attacks. The interviewers generally agreed that displaying their badges helped to gain cooperation from respondents. The field interviewers in these two focus groups also stated, however, that they faced some additional challenges as a result of September 11. In one incident, a respondent asked that a police officer be present during the scheduled home visit.

Other observations by the 11 interviewers include these:

• None of the interviewers noticed concerns related to the anthrax scare.
• Interviewers received numerous questions or comments from recipients regarding the number of agencies and organizations identified by the survey. The lead letter uses a Department of Health and Human Services letterhead while interviewers identify themselves as being from RTI, representing the U.S. Public Health Service. Letters use different return addresses and postmarks.
• In the first few weeks after September 11, interviewers believed people were more willing to participate. The effect, however, seemed to fade after about a month.
• Some interviewers wore flags on their badges after September 11 and believed badges encouraged cooperation.
• Some interviewers felt foreign-born respondents were more hesitant to cooperate. Interviewers said these respondents may have been more fearful of being deported or thought the government was investigating to see if they were associated with terrorists. However, some respondents from the Middle East seemed more willing to cooperate and wanted to make statements supporting America.
• In a town near a military base, people were generally more hesitant to participate or suspicious following the attacks.
• No difficulties or differences in travel were mentioned by these focus groups.

Conclusions. The willingness of respondents to participate may have generally increased following the September 11 attacks, although specific groups may have been more reluctant to participate. However, field interviewers viewed these effects as temporary.

Interviewers generally believe the quality of their identification materials (badge and lead letter) can significantly affect respondent confidence. Better, larger badges seem to encourage participation, while identifying several agencies and organizations within the lead letter and verbal introductions may confuse some respondents, discouraging them from participating.

Only the New York area interviewers noticed concerns among respondents resulting from anthrax letters. Some New York and Washington field interviewers indicated that traveling and parking were more difficult as a result of stricter security following the September 11 attacks, while none of the interviewers outside the metropolitan areas noticed changes in travel or parking conditions.

¹Alabama, California, Illinois, Indiana, Michigan, Montana, Pennsylvania, Texas, and Utah.

This page was last updated on June 16, 2008.

SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States.