ISCAM: Investigating Statistical Concepts, Applications, and Methods

In this chapter, you have again focused on comparing the outcomes for different groups, but for quantitative data. The tools for descriptive statistics are the same as in Chapter 2 (e.g., means, medians, standard deviations, interquartile ranges, dotplots, histograms, and boxplots).

After exploring and comparing the groups descriptively, it may be appropriate to ask whether the differences observed could have arisen "by chance alone." In other words, is the observed difference large enough to convince us that it arose from a genuine difference in the groups instead of from the randomness inherent in designing the experiment or in selecting the samples? Analogous to comparing proportions in Chapter 3, we used simulation to approximate how often we would obtain a difference in means at least as extreme as observed just by chance (random sampling or random assignment). Then we learned of a "large sample" approach we can use in either case to approximate the null distribution of the standardized statistic, yielding the two-sample t-test. Keep in mind that the logic of statistical significance hasn’t changed – we assume there is no effect or no population difference and determine how often we would get a sample difference result at least as extreme as what was observed. You saw that this chance is strongly affected by the amount of variability within the groups. It is very important when reporting on your statistical study that you also mention the sample sizes. This helps you evaluate the power and practical significance of the study results. You were also reminded that no matter how small a p-value is, we cannot draw cause-and-effect conclusions unless the data came from a properly designed randomized experiment.

You will often have the choice between a randomization-based test or the t-procedures. The advantage to the randomization procedures is they allow us to work with many types of statistics (such as median and trimmed mean) where we do not have theoretical results on the large sample behavior of the randomization distribution. The advantage of the t-procedures is the ease in calculating confidence intervals. It is always important to consider the technical conditions of an inference procedure before applying the procedure (e.g., make sure you really have two independent samples). If the conditions are not met, transformations may be one way of rescaling the data to be more appropriate for t procedures. If data are paired, then you should consider one-sample procedures on the differences. Such pairing is often very useful in increasing the efficiency of the study. These t-procedures are quite robust in calculating p-values and confidence intervals.

Always keep in mind the importance of considering whether there is random sampling and/or random assignment in the study design and how that impacts the scope of conclusions that you can draw from your study.

Note: You can also consider transforming the data before applying normal-based methods.