Python QQ和PP两个不等长的分布图

Question

I am not sure what the best/most statistically sound way to accomplish what I want is, but I am basically trying to take a distribution of p-values and compare it to a much larger distribution of p-values created by permuting my original data. I am working with small p-values, so I am actually comparing the log10 of the p-values.

I have been trying to figure out a good general way to compare two arrays with similar values but unequal lengths. What I really want is something like scipy.qqplot(dataset1, dataset2) , but that doesn't exist, the QQ plot only compares your distribution to an established distribution (this question has been asked for R also: https://stats.stackexchange.com/questions/12392/how-to-compare-two-datasets-with-qq-plot-using-ggplot2 ).

Essentially this amounts to comparing two histograms. I can use np.linspace to force the exact same bins for each distribution:

bins = 100
mx = max(np.max(vector1), np.max(vector2))
mn = min(np.min(vector2), np.max(vector2))
boundaries = np.linspace(mn, mx, bins, endpoint=True)
labels = [(boundaries[i]+boundaries[i+1])/2 for i in range(len(boundaries)-1)]

I can then easily use these boundaries and labels to make two histograms, weighted by the length of the original vectors. The easiest thing to do is just just use a few bins and plot them as histograms on the same axis, like in this question:

However, I really want something more like a QQ plot, and I want to use a lot of bins, so that I can see even small deviations from the 1-to-1 line. The problem with just plotting the two histograms, is that they look like this:

The two plots are just right on top of each other, I can't see anything.

So what I want to figure out, is how to compare these two histograms while maintaining the bin labels . I can easily plot the two against each other as a scatter graph, but that ends up being indexed by the bin frequency:

What I really want, is to just compare the two histograms, or to make a QQ plot of the differences, but I cannot come up with a good statistically sound way of doing this. I can find no methods that allow me to make a QQ plot with two datasets instead of one dataset and a built in distribution, and I can't find any way of plotting two distributions of unequal length against each other.

For reference, here are the two histograms that went into creating that plot, you can see that they are extremely similar:

I know there must be a good way of doing this, because it seems so obvious, but I am new to this kind of thing, and relatively new to scipy, pandas, and statsmodels also.

I intentionally have not provided an example distribution here, because I wasn't sure how to make a minimal set of arrays that were non-normally distributed and captured what I am trying to do; plus the point is to be able to do this for any two overlapping unequal-length arrays.

What I want to know is what is the right/best way to approach this problem in python in a statistically sound way? Is there some way of creating a distribution from the permuted data that could be used for a statsmodels or scipy QQ plot? Is there a way to compare two histograms visually like this already? Is there a way of making probability plots that I don't know about?

---

Edit: Trying cumulative and manual QQ plots

Thanks to @user333700's answer, I figured out how to create a manual QQ plot for the data, and also a cumulative probability plot. I created the plots using data with an overlapping min/max but the following distributions:

QQ plot:

q = np.linspace(0, 100, 101)
fig, ax = plt.subplots()
ax.scatter(np.percentile(ytest, q), np.percentile(xtest, q))

So that works really well with simple data, the cumulative plot is similar:

# Pick bins
x = ytest
y = xtest
boundaries = sorted(x)[::round(len(x)/bins)+1]
labels = [(boundaries[i]+boundaries[i+1])/2 for i in range(len(boundaries)-1)]

# Bin two series into equal bins
xb = pd.cut(x, bins=boundaries, labels=labels)
yb = pd.cut(y, bins=boundaries, labels=labels)

# Get value counts for each bin and sort by bin
xhist = xb.value_counts().sort_index(ascending=True)/len(xb)
yhist = yb.value_counts().sort_index(ascending=True)/len(yb)

# Make cumulative
for ser in [xhist, yhist]:
    ttl = 0
    for idx, val in ser.iteritems():
        ttl += val
        ser.loc[idx] = ttl

# Plot it
fig, ax = plt.subplots(figsize=(6,6))
ax.scatter(xhist, yhist)
plt.show()

Going back to my actual skewed data (where the two distributions are extremely similar in every way except the lengths) and adding a 1-to-1 line, I get this for those two:

So both work, which is great, and the cumulative probability plot shows quite clearly that there is no large difference in the data, but the QQ plot shows that there is a small difference in the tail.

Answer 1

In terms of statistical tests, scipy has a two sample Kolmogorov-Smirnov test for the continuous variables. The binned histogram data can be used with a chisquare test. scipy.stats also has a k-sample Anderson-Darling test.

For plotting:

The equivalent of a probability plot for two histograms would be to plot the cumulative frequencies for the two samples, ie with cumulative probabilities on each axis corresponding to the bin boundaries.

statsmodels has a qq-plot for two sample comparison, however it currently assumes that the sample sizes are the same. If the sample sizes are different, then the quantiles need to be computed for the same probabilities. https://github.com/statsmodels/statsmodels/issues/2896 https://github.com/statsmodels/statsmodels/pull/3169 (I don't remember what the status of this is.)