Python Numpy: How does numpy.exp(x) coerce the return value of this sigmoid function into an ndarray?

Question

I'm looking at a preliminary neural network example using the numpy package in python, and I am a little confused about what appears to be some magical type coercion.

Take this sigmoid function as reference, it uses numpy's numpy.exp(x) to calculate e to the power of x:

def sigmoid(z):
    return 1.0 / (1.0 + numpy.exp(-z))

This is very straight forward - it looks like it should return a float, and indeed does so when fed a float or integer as argument:

sigmoid(0) # => 0.5

What is strange to me is that (according to some footnotes on numpy) when passed a numpy ndarray this function will return a numpy ndarray with the sigmoid function applied to each element:

sigmoid(numpy.random.randn(2,2)) # => 2x2 ndarray 

This makes very little sense to me as the function clearly returns a float divided by a statement containing the numpy functions. How can numpy coerce the return value like this? How is this even possible?

Is their something very strange about python functions that I'm missing? Can numpy somehow change the return property of the function object it executes inside of?

Answer 1

Given a scalar, your function returns a float wrapped in a numpy type:

In [136]: sigmoid(0)
Out[136]: 0.5
In [137]: type(_)
Out[137]: numpy.float64

Given an array, it returns an array of the same shape (but float dtype):

In [140]: sigmoid(np.arange(3))
Out[140]: array([0.5       , 0.73105858, 0.88079708])

np.exp does that.

The + and / produce a numpy object if one of the arguments is an numpy object. The interpreter translates the + into a call to __add__ or __radd__ . There are various rules when the arguments are different types, but in most cases, if either argument is numpy array, the calculation is controlled by the numpy version of __add__ .

So in 1.0 / (1.0 + numpy.exp(-z)) , the np.exp produces an array, and the rest of the calculation follows suit.

And the normal behavior of math calculations involving numpy arrays is to apply the calculation to each element of the array(s). We say they operate element-wise. It gets more complicated when working with several arrays that differ in shape ; then broadcasting rules apply.

There are also numpy operations that combine elements of an array, eg np.sum , and ones that perform more complication combinations like matrix multiplication.

The math module version of exp only works with Python scalars:

In [141]: import math
In [142]: 1.0 / (1.0 + math.exp(-0))
Out[142]: 0.5
In [143]: type(_)
Out[143]: float
In [144]: 1.0 / (1.0 + math.exp(-np.arange(3)))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-144-fa5530a5ba95> in <module>()
----> 1 1.0 / (1.0 + math.exp(-np.arange(3)))

TypeError: only size-1 arrays can be converted to Python scalars