在每个像元中创建具有相同数量点的2D历史

Question

I am looking for a possibility to create a 2D histogram with irregular bin sizes with the possibility of plotting heat as the z variable.

The data: I have one billion objects. Every object has the features x, y and an anomaly score of z.

The plot: Plotted are all objects with y against x. The histogram should have irregular (adaptive) bin sizes, so that in every bin created lie the same amount of objects. This should initially create a histogram without any visible traits, having just one color (with color representing the count of objects).

To create the bin edges I firstly use np.percentiles and separate the objects based on the x feature into percentiles. Secondly, I use the first x binedge, find all the points within it and bin them in the y direction based on percentiles. That would look something like this (pseudocode):

for i, key_x in enumerate(np.percentile(x, np.arange(0,101, 10))):
    xedges[i] = key_x
    objects = find_all_objects_within_binedge(key_x)

    for j, key_y in enumerate(np.percentile(objects["y"], np.arange(0,101, 10))):
        yedges[i, j] = key_y

So xedges is an array with the binedges in x direction and yedges is a matrix giving me the y binedges for every x binedge. If this is not understandable please let me know.

So if we imagine the histogram that would result, we would have straight binning lines in x. But in the y direction these lines would be split. See here to get an idea of what I mean with the y bins being irregular split.

And this is were I am stuck. I have no idea how to create a histogram or plot from my x-binedges and y-binedges with these irregular bins.

The goal (for better understanding): Once that is accomplished, I would like to be able to have each bin colored by the mean or std of all the points within that cell using the z values (have the code for that ready). Ideally this will look very smooth as well, with some minor exceptions, which would be anomalous and what I am looking for. But this should be feasible with plt.pcolormesh.

English is not my native language and I tried my best to describe the problem. If something is unclear please let me know and I'll try to clarify as best as possible. Thank you guys in advance :)

Answer 1

From what I understand you want the data to be binned based on equal amounts of data in the bin. Indeed percentiles can be used for this purpose. If you use numpy you can do this along d dimensions. Here is an example for 2d binning:

import matplotlib.pyplot as plt
from numpy import array, random, percentile

data = random.randn(1000, 2)
data[:, 1] = data[:, 1] * .1 + 1 # shift the gauss


percentiles = percentile(data, range(0, 100, 10), axis = 0)

fig, ax = plt.subplots()
ax.hist2d(*data.T, bins = percentiles.T)
fig.show()

Is this what you were looking for?

Edit: non-uniform grid example

import matplotlib.pyplot as plt
from numpy import *
data = random.randn(1000, 2)
data[:, 1] = data[:, 1] * .1 + 1 # shift the gauss

xper = percentile(data[:, 0], range(0, 101, 10))
yper = zeros((xper.size, xper.size))

binnedData = ones(yper.shape)
for index, (binstart, binend) in enumerate(zip(xper[:-1], xper[1:])):
    idx = where(logical_and(data[:, 0] >= binstart, data[:, 0] <= binend))[0] # expensive
    yper[index] = percentile(data[idx, 1], range(0, 101, 10))
    for jndex, j in  enumerate(digitize(data[idx, 1], yper[index])):
        j -= 1 #digit takes right bins
        # generate dummy values
        binnedData[index, j] += data[idx[j], :].sum() /  xper.size
fig, ax = plt.subplots()
ax.pcolormesh(xper, yper, binnedData)

Answer 2

It seems the question asks for a way to plot values on a grid, which is regular in one dimension, but irregular in the other.
As I understand it such grid would be defined by a 1D array in eg x-direction, and a 2D array in y-direction. Both arrays would denote the edges of the grid cells in the respective dimension.

For a M x N grid, x_edges would hence have N+1 elements, and y_edges would be of shape (M+1, N) . The following would be a 4 x 3 grid.

x_edges = np.array([0,1,2,3])
y_edges = np.array([[0.,0.,0.],
                    [.3,.2,.2],
                    [.5,.6,.4],
                    [.8,.9,.7],
                    [1.,1.,1.]])

The usual matplotlib tools like imshow or pcolor do - as far as I can see - not allow to plot such grids. An alternative is hence to use a PolyCollection and plot the respective rectangles with it.

An array of values that shall be mapped to color can be set to that collection. This array should have one value less per dimension and be flat, ie have M*N elements.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.collections import PolyCollection

# Starting data: A grid, regular in x-direction and irregular in y direction.
x_edges = np.array([0,1,2,3])
y_edges = np.array([[0.,0.,0.],
                    [.3,.2,.2],
                    [.5,.6,.4],
                    [.8,.9,.7],
                    [1.,1.,1.]])

######## Grid creation ################
#y_edges = np.concatenate((y_edges, np.zeros(len(y_edges))))
s = np.array(y_edges.shape)
# make x_edges 2D as well.
x_edges = np.tile(x_edges, s[0]-1).reshape((s[0]-1, s[1]+1))

# you may also have an array of values. 
# This should be of shape one less than the edges and flattened.
values = np.arange(np.prod(s+np.array((-1,0))))

# Produce a vertices array of the edges of rectangles that form each pixel.
x = np.c_[x_edges[:,:-1].flatten(), x_edges[:,:-1].flatten(),
          x_edges[:,1: ].flatten(), x_edges[:,1: ].flatten()]
y = np.c_[y_edges[:-1,:].flatten(), y_edges[1: ,:].flatten(),
          y_edges[1: ,:].flatten(), y_edges[:-1,:].flatten()]
xy = np.stack((x,y), axis=2)

# Create collection of rectangles.
pc = PolyCollection(xy, closed=True, edgecolors="k", linewidth=0.72, cmap="inferno")
pc.set_array(values)

######## Plotting ################
fig, ax = plt.subplots()
ax.add_collection(pc)
fig.colorbar(pc, ax=ax)

ax.margins(0)
ax.autoscale()
plt.show()

This grid uses a small number of cells to show the principle. If you want to have more cells, make sure not to plot the edges of rectangles by removing the edgecolors and linewidth arguments.