How to make curvilinear plots in matplotlib

Question

I am trying to visualize how two-dimensional data gets transformed and "bent" as it passes through the layers of a neural network. Affine transformations and translations are easy, but visualizing how an activation function (such as tanh or the logistic function) bends 2D space into a curvilinear grid is posing more of a challenge.

To see what I mean, Chris Olah has done exactly this in his post Neural Networks, Manifolds, and Topology .

Do any of you guys know how to do this?

Answer 1

I ended up with the following solution:

First, I specified a regular 2D grid using the NumPy linspace function:

x_range = range(-5,6)
y_range = range(-5,6)

lines = np.empty((len(x_range)+len(y_range), 2, 100))

for i in x_range: # vertical lines
    linspace_x = np.linspace(x_range[i], x_range[i], 100)
    linspace_y = np.linspace(min(y_range), max(y_range), 100)
    lines[i] = (linspace_x, linspace_y)
for i in y_range: # horizontal lines
    linspace_x = np.linspace(min(x_range), max(x_range), 100)
    linspace_y = np.linspace(y_range[i], y_range[i], 100)
    lines[i+len(x_range)] = (linspace_x, linspace_y)

Then, I performed an arbitrary affine transformation on the grid. (This mimics the vector-matrix multiplication between activations and weights in a neural net.)

def affine(z):
    z[:, 0] = z[:, 0] + z[:,1] * 0.3 # transforming the x coordinates
    z[:, 1] = 0.5 * z[:, 1] - z[:, 0] * 0.8 # transforming the y coordinates
    return z

transformed_lines = affine(lines)

Last but not least, using the (now transformed) coordinates making up each line in the grid, I applied a nonlinear function (in this case the logistic function):

def sigmoid(z):
    return 1.0/(1.0+np.exp(-z))

bent_lines = sigmoid(transformed_lines)

Plotting the new lines using matplotlib:

plt.figure(figsize=(8,8))
plt.axis("off")
for line in bent_lines:
    plt.plot(line[0], line[1], linewidth=0.5, color="k")
plt.show()

The result: