Relative insensitivity to sample sizes in judgments of frequency distributions : people are similarly confident in the results from 30 versus 3,000 observations

Abstract

Six experiments test whether people are sensitive to variations in the sample size or relatively insensitive to sample sizes varying by one or two orders of magnitude. We posit that past studies found that people are increasingly confident in the sample mean as the sample size increases because variations in the sample size were likely highlighted by the within-participant designs used in many of these studies. Using between-participant designs, we find that people are only slightly sensitive to variations in the sample size by a factor of 50, 100, and 400 when they are making confidence judgments. Our finding suggests that the psychological law of small numbers applies not only to people’s judgments about sample variances but also to their judgments of sample means. An intervention that provided people with the results of a statistical test about the extent to which the data are consistent with the null hypothesis versus the alternate hypothesis helped reduce people’s insensitivity to variations in the sample size.