ISCAM: Investigating Statistical Concepts, Applications, and Methods

In this chapter, you have focused on making inferences based on a representative sample from a random process that can be repeated indefinitely under identical conditions or from a finite population, for a single binary variable. You have learned how to model the chance variation in the binary outcomes that arise from such a sampling process.

In each case, when the p-value is small, we have evidence that the observed result did not happen solely due to the random chance inherent in the sampling process (a "statistically significant result"). The smaller the p-value, the stronger the evidence against the null hypothesis.

Still, we must keep in mind that we are merely measuring the strength of evidence – we may be making a mistake whenever we decide whether or not to reject a null hypothesis. If we decide the observed result did not happen by chance and so reject the null hypothesis, there is still a small probability that the null hypothesis is true and that the observed result did occur by chance (the probability of committing a Type I error). If we decide the result did happen by chance, there is still a probability that something other than random chance was involved (a Type II error). It is important to consider the probabilities of these errors when completing your assessment of a study. In particular, the sample size of the study can influence the probability of a Type II error and the related idea of power, which is the probability of correctly rejecting a null hypothesis when it is false.

You began your study of confidence intervals as specifying an interval of plausible values for the process probability based on what you observed in the sample. These were the hypothesized parameter values that generated two-sided p-values above the level of significance \(\alpha\text{.}\) In other words, they were the parameter values for which your sample result would not be surprising. When the sample size is large (large enough for the normal approximation to be valid), we saw that these confidence intervals have a very convenient form: statistic ± margin-of-error where margin-of-error = critical value × standard error of statistic. The critical value is the number of standard errors you want to use corresponding to a specified confidence level. Keep in mind that the level of confidence provides a measure of how reliable the procedure will be in the long-run (which can vary by procedure and sample conditions).

Finally, you saw that this reasoning process holds equally well when the sampling is from a finite population, where the randomness in our model comes from the selection of the observational units, not from the observational units’ individual outcomes. In which case, use of random sampling allows us to believe that our sample is representative of (has similar characteristics as) the larger population. [But still be on the look out for possible non-sampling errors (e.g., wording of a question).] Use of simple random samples also allows us to estimate the sample-to-sample variation in our statistic. Technically we should use the hypergeometric distribution to model the behavior of the statistic. But if the population is large compared to the size of the sample (e.g., more than 20 times larger), then we can still use the binomial distribution; and if the sample size is also large we can use normal-based methods to determine p-values and confidence intervals (as well as power and sample size calculations). The interpretation of the p-value is essentially the same but now applies to the randomness inherent in the sampling process. Also keep in mind that the confidence interval aims to capture the proportion of the population having the characteristic of interest (which is equivalent to the probability of selecting one individual from the population with that characteristic when the population is large).