Employment Discrimination and Statistics: What Does Statistical Significance Mean?

One of the most important pieces of information in inferential statistics is the "statistical significance” of the results. Statistical significance refers to how likely it is that the relationship you found between the variables might be due to chance.

For example, imagine you found a correlation of .33 between age and pay. Most computer software programs will automatically calculate the statistical significance or “p value” (where “p” stands for probability) of this correlation. Over the years, social scientists have developed a convention that a p-value of .05 or less is a sufficiently small enough number to conclude that there is a relationship between age and pay. In other words, if the probability is 5% or less that this relationship is due to chance, then one can safely conclude that there is a relationship between the two variables. If however the p-value is greater than 5% (say, for example, .10 or 10%), then we should conclude that there is not a relationship between age and pay.

It should be noted that in social sciences today, the statistical significance test is being critiqued for several reasons. A major reason for criticism is that statistical significance is dependent, in part, on the size of the sample size. That is, the larger the sample size, the lower the p-value will be, all things being equal. One could have a situation, then, where the relationship between two variables is very small (a correlation of .09, for example), but is statistically significant because of the large sample size.

What statistical significance does NOT mean…..

1. statistical significance does not necessarily mean that the two variables have a practically important relationship;

2. statistical significance does not mean that one variable necessarily causes the other variable.

What have the courts said? In Castaneda v. Partida, 430 U.S. 482 (1977), the Supreme Court referred to a rule of “two or three standard deviations,” which corresponds to p-values of .05 (about two standard deviations above the mean) to .01 (roughly three standard deviations above the mean). Castaneda v. Partida is often cited in subsequent discrimination cases, in determining whether the relationship between two variables is meaningful.

In Hazelwood School District v. United States, 433 U.S. 299 (1977), for example, the Supreme Court in 1977 examined the difference between the number of African-American teachers in the Hazelwood School District compared to the number of African-American teachers in the relevant labor market and used the rule of two or three standard deviations to assess statistical significance.

Although the courts have generally relied on the “two or three standard deviations” rule, things can be much more complicated. Let’s look at the recent Dukes v. Wal-Mart ruling regarding class certification.

On page 37, in footnote number 33, the judge noted that the plaintiffs’ expert Dr. Drogin shows that while women received fewer promotions to Co-Manager in 37 of 40 regions in the country, “the disparity was of a statistically significant value in only 22 regions.” Even more critically, while women received fewer Store Manager promotions in 34 of 40 regions, the difference was statistically significant in only “13 regions.” Nonetheless, the judge accepted the discrepancies as evidence of discrimination by the plaintiffs, based on among other arguments, the “wide-spread discrimination at the lower levels carries through to the upper levels…”