遍历列表的最有效方法

Question

I'm currently collecting data from quandl and is saved as a list of lists. The list looks something like this (Price data):

['2', 1L, datetime.date(1998, 1, 2), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), '82.1900', '83.6200', '81.7500', '83.5000', '28.5183', 1286500.0]

This is typically 1 of about 5000 lists, and every once in awhile Quandl will spit back some NaN values that don't like being saved into the database.

['2', 1L, datetime.date(1998, 1, 2), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), 'nan', 'nan', 'nan', 'nan', 'nan', 0]

What would be the most efficient way of iterating through the list of lists to change 'nan' values into zeros?

I know I could do something like this, but it seems rather inefficient. This operation will need to be performed on 11 different values * 5000 different dates * 500 companies:

def screen_data(data):
    new_data = []
    for d in data:
        new_list = []
        for x in d:
            new_value = x
            if math.isNan(x):
                new_value = 0
            new_list.append(new_value)

        new_data.append(new_list)
    return new_data

I would be interested in any solution that could reduce the time. I know DataFrames might work, but not sure how it would solve the NaN issue.

Or if there is a way to include NaN values in an SQLServer5.6 database along with floats, changing the database is also a viable option.

Answer 1

Don't create a new list - rather, edit the old list in-place:

import math

def screenData(L):
    for subl in L:
        for i,n in enumerate(subl):
            if math.isnan(n): subl[i] = 0

The only way I can think of, to make this faster, would be with multiprocessing

Answer 2

I haven't timed it but have you tried using nested list comprehension with conditional expressions ?

For example:

import datetime

data = [
    ['2', 1, datetime.date(1998, 1, 2),
     datetime.datetime(2016, 9, 26, 1, 35, 3, 830563),
     datetime.datetime(2016, 9, 26, 1, 35, 3, 830563),
     '82.1900', '83.6200', '81.7500', '83.5000',
     '28.5183', 1286500.0],
    ['2', 1, datetime.date(1998, 1, 2),
     datetime.datetime(2016, 9, 26, 1, 35, 3, 830563),
     datetime.datetime(2016, 9, 26, 1, 35, 3, 830563),
     'nan', 'nan', 'nan', 'nan', 'nan', 0],
]

new_data = [[y if str(y).lower() != 'nan' else 0 for y in x] for x in data]

print(new_data)

I did not use math.isnan(y) because you have to be sure that y is a float number or you'll get an error. This is much more difficult to do while almost everything has a string representation. But I still made sure that I did the lower case comparison to 'nan' (with .lower() ) since 'NaN' or 'Nan' are legal ways to express "Not a Number".

Answer 3

how about this

import math

def clean_nan(data_list,value=0):
    for i,x in enumerate(data_list):
        if math.isnan(x):
            data_list[i] = value
    return data_list 

(the return is optional, as the modification was made in-place, but it is needed if used with map or similar, assuming of course that data_list is well a list or similar container)

depending on how you get your data and how you work with it will determined how to use it, for instance if you do something like this

for data in (my database/Quandl/whatever):
    #do stuff with data

you can change it to

for data in (my database/Quandl/whatever):
    clean_nan(data)
    #do stuff with data

or use map or if you are in python 2 imap

for data in map(clean_nan,(my database/Quandl/whatever)):
    #do stuff with data

that way you get to work with your data as soon as that arrive from the database/Quandl/whatever, granted if the place where you get the data also work as a generator, that is don't process the whole thing all at once, and if it does, procure to change it to a generator if possible. In either case with this you get to work with your data as soon as possible.