How do you check data distribution in python?

Many methods exist for testing whether a variable has a normal distribution. In this article, you will find out which one to use!

1. Histogram

1.1. Introduction

The first method that almost everyone knows is the histogram. The histogram is a data visualization that shows the distribution of a variable. It gives us the frequency of occurrence per value in the dataset, which is what distributions are about.

The histogram is a great way to quickly visualize the distribution of a single variable.

1.2. Interpretation

In the picture below, two histograms show a normal distribution and a non-normal distribution.

• On the left, there is very little deviation of the sample distribution (in grey) from the theoretical bell curve distribution (red line).
• On the right, we see quite a different shape in the histogram, telling us directly that this is not a normal distribution.

Sometimes the deviation from a normal distribution is so obvious that it can be detected visually.

1.3. Implementation

A histogram can be created easily in python as follows:

Creating a histogram using pandas in python

1.4. Conclusion

The histogram is a great way to quickly visualize the distribution of a single variable.

2. Box Plot

2.1. Introduction

The Box Plot is another visualization technique that can be used for detecting non-normal samples. The Box Plot plots the 5-number summary of a variable: minimum, first quartile, median, third quartile and maximum.

The boxplot is a great way to visualize distributions of multiple variables at the same time.

2.2 Interpretation

The boxplot is a great visualization technique because it allows for plotting many boxplots next to each other. Having this very fast overview of variables gives us an idea of distribution and as a “bonus”, we get the complete 5-number summary that will help us in further analysis.

You should look at two things:

• Is the distribution symmetrical (as is the Normal distribution)?
• Does the width (opposite of pointiness) correspond to the width of the normal distribution? This is hard to see on a box plot.

Normal (left), Uniform (middle) and exponential (right) boxplots vs normal bell curve

2.3. Implementation

A boxplot can be easily implemented in python as follows:

Creating a boxplot using pandas in python

2.4. Conclusion

The boxplot is a great way to visualize distributions of multiple variables at the same time, but a deviation in width/pointiness is hard to identify using box plots.

3. QQ Plot

3.1. Introduction

With QQ plots we’re starting to get into the more serious stuff, as this requires a bit more understanding than the previously described methods.

QQ Plot stands for Quantile vs Quantile Plot, which is exactly what it does: plotting theoretical quantiles against the actual quantiles of our variable.

The QQ Plot allows us to see deviation of a normal distribution much better than in a Histogram or Box Plot.

3.2. Interpretation

If our variable follows a normal distribution, the quantiles of our variable must be perfectly in line with the “theoretical” normal quantiles: a straight line on the QQ Plot tells us we have a normal distribution.

Normal (left), uniform (middle) and exponential (right) QQ Plots

As seen in the picture, the points on a normal QQ Plot follow a straight line, whereas other distributions deviate strongly.

• The uniform distribution has too many observations in both extremities (very high and very low values).
• The exponential distribution has too many observations on the lower values, but too little in the higher values.

In practice, we often see something less pronounced but similar in shape. Over or underrepresentation in the tail should cause doubts about normality, in which case you should use one of the hypothesis tests described below.

3.3. Implementation

Implementing a QQ Plot can be done using the statsmodels api in python as follows:

Creating a QQ Plot using statsmodels

3.4. Conclusion

The QQ Plot allows us to see deviation of a normal distribution much better than in a Histogram or box plot.

4. Kolmogorov Smirnov test

4.1. Introduction

If the QQ Plot and other visualization techniques are not conclusive, statistical inference (Hypothesis Testing) can give a more objective answer to whether our variable deviates significantly from a normal distribution.

If you have doubts about how and when to use hypothesis testing, here’s an article thatgives an intuitive explanation to hypothesis testing.

The Kolmogorov Smirnov test computes the distances between the empirical distribution and the theoretical distribution and defines the test statistic as the supremum of the set of those distances.

The advantage of this is that the same approach can be used for comparing any distribution, not necessary the normal distribution only.

The KS test is well-known but it has not much power. It can be used for other distribution than the normal.

4.2. Interpretation

The Test Statistic of the KS Test is the Kolmogorov Smirnov Statistic, which follows a Kolmogorov distribution if the null hypothesis is true.

If the observed data perfectly follow a normal distribution, the value of the KS statistic will be 0. The P-Value is used to decide whether the difference is large enough to reject the null hypothesis:

• If the P-Value of the KS Test is larger than 0.05, we assume a normal distribution
• If the P-Value of the KS Test is smaller than 0.05, we do not assume a normal distribution

4.3. Implementation

The KS Test in Python using Scipy can be implemented as follows. It returns the KS statistic and its P-Value.

Applying the KS Test in Python using Scipy

4.4. Conclusion

The KS test is well-known but it has not much power. This means that a large number of observations is necessary to reject the null hypothesis. It is also sensitive to outliers. On the other hand, it can be used for other types of distributions.

5. Lilliefors test

5.1. Introduction

The Lilliefors test is strongly based on the KS test. The difference is that in the Lilliefors test, it is accepted that the mean and variance of the population distribution are estimated rather than pre-specified by the user.

Because of this, the Lilliefors test uses the Lilliefors distribution rather than the Kolmogorov distribution.

Unfortunately for Lilliefors, it’s power is still lower than the Shapiro Wilk test.

5.2. Interpretation

• If the P-Value of the Lilliefors Test is larger than 0.05, we assume a normal distribution
• If the P-Value of the Lilliefors Test is smaller than 0.05, we do not assume a normal distribution

5.3. Implementation

The Lilliefors test implementation in statsmodels will return the value of the Lilliefors test statistic and the P-Value as follows.

Attention: in the statsmodels implementation, P-Values lower than 0.001 are reported as 0.001 and P-Values higher than 0.2 are reported as 0.2.

Applying the Lilliefors test using statsmodels

5.4. Conclusion

Although Lilliefors is an improvement to the KS test it’s power is still lower than the Shapiro Wilk test.

6. Shapiro Wilk test

6.1. Introduction

The Shapiro Wilk test is the most powerful test when testing for a normal distribution. It has been developed specifically for the normal distribution and it cannot be used for testing against other distributions like for example the KS test.

The Shapiro Wilk test is the most powerful test when testing for a normal distribution.

6.2. Interpretation

• If the P-Value of the Shapiro Wilk Test is larger than 0.05, we assume a normal distribution
• If the P-Value of the Shapiro Wilk Test is smaller than 0.05, we do not assume a normal distribution

6.3. Implementation

The Shapiro Wilk test can be implemented as follows. It will return the test statistic called W and the P-Value.

Attention: for N > 5000 the W test statistic is accurate but the p-value may not be.

Applying the Shapiro Wilk test using statsmodels in Python

6.4. Conclusion

The Shapiro Wilk test is the most powerful test when testing for a normal distribution. You should definitely use this test.

7. Conclusion — which approach to use!

For quick and visual identification of a normal distribution, use a QQ plot if you have only one variable to look at and a Box Plot if you have many. Use a histogram if you need to present your results to a non-statistical public.

As a statistical test to confirm your hypothesis, use the Shapiro Wilk test. It is the most powerful test, which should be the decisive argument.

When testing against other distributions, you cannot use Shapiro Wilk and should use for example the Anderson-Darling test or the KS test.

How do you check for data distribution in Python?

How do I know what distribution My data is?

How do you check a column distribution in Python?

How do you test a distribution?