numpy.histogram: problems with empty cells in excel

Question

I'm new using python so I don't know if I get all the technical terms right.

I'm using xlrd to read data from an excel-sheet, then I filter it with a filter function and then I create a histogram with the numpy.histogram function. Now I have an empty cell in the excel-sheet and the numpy.histogram gives back wrong results:

this is my code:

import xlrd
import openpyxl
import numpy as n
from numpy import *   

file_location = "C:/Users/test.xlsx"
sheet_index = 2
range_hist = 23
lifetime_data = 3
low_salesyear = 1990
upp_salesyear = 2005
col_filter1 = 14
filter_value1 = 1
col_filter2 = 18
filter_value2 = 5


    # open excel-file
    workbook = xlrd.open_workbook(file_location)


    # get sheet, index always start at 0
    sheet = workbook.sheet_by_index(sheet_index)


    #read all data in the sheet
    list_device = [[sheet.cell_value(r,c) for c in range (sheet.ncols)] for r in range (1,sheet.nrows)]


    # filter list for independent variables
    listnew = list(filter(lambda x:  x[col_filter1]==filter_value1 and x[col_filter2]==filter_value2 and low_salesyear <= x[0] <= upp_salesyear, list_device))
    # low_salesyear <= x[0] <= upp_salesyear and


    # select relevant data from filtered list for histogram and store it in list for histogram

    list_for_hist = []
    for i in range(len(listnew)):
        list_for_hist.append(listnew[i][lifetime_data])
    print (list_for_hist)


    # create array from list
    array_for_hist = array(list_for_hist)


    # create histogram
    hist = np.histogram(array_for_hist, bins = range(0,int(range_hist)))
    print (hist)

I put all the variables in the beginning so I can easily change them. I'm sure there would be a more elegant way to program the whole thing.

The list I'm filtering from excel looks like this:

[8.0, 19.0, 4.0, 4.0, 8.0, 3.0, 13.0, '', 10.0, 7.0, 17.0, 16.0, 8.0,
6.0, 13.0, 8.0, 7.0, 11.0, 12.0, 13.0, 4.0, 6.0, 5.0, 19.0, 8.0, 6.0]

The resulting hist from the numpy.histogram looks like this:

(array([  0,  10,   0,   1,   3,   1,   3,   2,   5, -25,   1,   1,   1,
         3,   0,   0,   1,   1,   0,   2,   0,   0]), array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
       17, 18, 19, 20, 21, 22]))

So I don't understand why it gives back 10 for the bin 1 and -25 for the bin 9. If I eliminate the empty cell in excel, the histogram gets right.

Would there be a way to tell my program to ignore empty cells?

Thank you so much for your help!

Answer 1

np.array(list_for_hist) converts all items in list_for_hist to a common dtype. When list_for_hist contains both floats and strings, np.array returns an array containing all strings:

In [32]: np.array(list_for_hist)
Out[32]: 
array(['8.0', '19.0', '4.0', '4.0', '8.0', '3.0', '13.0', '', '10.0',
       '7.0', '17.0', '16.0', '8.0', '6.0', '13.0', '8.0', '7.0', '11.0',
       '12.0', '13.0', '4.0', '6.0', '5.0', '19.0', '8.0', '6.0'], 
      dtype='|S32')   <-- `|S32` means 32-byte strings.

So binning strings with bins=range(0,int(23)) probably should raise an exception, but instead np.histogram returns garbage.

You'll need to convert the list_for_hist to an array or list containing only floats:

import numpy as np
list_for_hist = [8.0, 19.0, 4.0, 4.0, 8.0, 3.0, 13.0, '', 10.0, 7.0, 17.0, 16.0,
                 8.0, 6.0, 13.0, 8.0, 7.0, 11.0, 12.0, 13.0, 4.0, 6.0, 5.0,
                 19.0, 8.0, 6.0]

array_for_hist = np.array(
    [item if isinstance(item,(float,int)) else np.nan for item in list_for_hist])
# create histogram
hist, bin_edges = np.histogram(array_for_hist, bins=range(0,int(23)))
print (hist)

yields

[0 0 0 1 3 1 3 2 5 0 1 1 1 3 0 0 1 1 0 2 0 0]