两次读取读取对象会导致“ IndexError：列表索引超出范围”

Question

When reading a csv reader object twice, getting the error 'IndexError: list index out of range'. Right now I'm creating a dictionary from iterating over the object, but fail when trying to create a similar list. Other code blocks are omitted for brevity, here is the pertinent code:

# Parse csv files for samples, creating a dictionary of key, value pairs and multiple lists.
with open('genes.csv') as f:
    cread = csv.reader(f, delimiter = '\t')
    sample_1_dict = {i: float(j) for i, j in cread}
    sample_1_list = [x for x in sample_1_dict.items()]
    sample_1_genes_sorted = sorted(sample_1_list, key=lambda expvalues: expvalues[0])
    sample_1_values_sorted = sorted(sample_1_list, key=lambda expvalues: expvalues[1])
    sample_1_genes = [i for i, j in sample_1_values_sorted]
    sample_1_values = [j for i, j in sample_1_values_sorted]
    sample_1_graph_un = [float(j) for i, j in cread]

...

sample_values_list = [i for i in sample_1_graph_un, sample_2_graph_un, sample_3_graph_un, sample_4_graph_un, sample_5_graph_un, sample_6_graph_un]

sample_graph_list_un = [[i for i in sample_value] for sample_value in sample_values_list]

colors = 'bgrcmy'
alphas = ['0.5', '0.5', '0.5', '0.5', '0.5', '0.5']
labels = ['278', '470', '543', '5934', '6102', '17163']

for graph, color, alpha, label in zip(sample_graph_list_un, colors, alphas, labels):
    plt.hist(graph, bins = 21, histtype = 'stepfilled', normed = True, color = color, alpha = float(alpha), label=label)

I'm resorting to reopening the csv file, and the following code does work:

# Parse csv files for samples, creating a dictionary of key, value pairs and multiple lists.
with open('genes.csv') as f:
    cread = csv.reader(f, delimiter = '\t')
    sample_1_dict = {i: float(j) for i, j in cread}
    sample_1_list = [x for x in sample_1_dict.items()]
    sample_1_genes_sorted = sorted(sample_1_list, key=lambda expvalues: expvalues[0])
    sample_1_values_sorted = sorted(sample_1_list, key=lambda expvalues: expvalues[1])
    sample_1_genes = [i for i, j in sample_1_values_sorted]
    sample_1_values = [j for i, j in sample_1_values_sorted]

...

with open('genes.csv') as f:
    cread = csv.reader(f, delimiter = '\t')
    sample_1_graph_un = [float(j) for i, j in cread]

sample_values_list = [i for i in sample_1_graph_un, sample_2_graph_un, sample_3_graph_un, sample_4_graph_un, sample_5_graph_un, sample_6_graph_un]

sample_graph_list_un = [[i for i in sample_value] for sample_value in sample_values_list]
colors = 'bgrcmy'
alphas = ['0.5', '0.5', '0.5', '0.5', '0.5', '0.5']
labels = ['278', '470', '543', '5934', '6102', '17163']

for graph, color, alpha, label in zip(sample_graph_list_un, colors, alphas, labels):
    plt.hist(graph, bins = 21, histtype = 'stepfilled', normed = True, color = color, alpha = float(alpha), label=label)

The difference in each code example is the location of the below statement in either one of the two 'with' blocks:

sample_1_graph_un = [float(j) for i, j in cread] 

Answer 1

You cannot read from a file, or a csv.reader() object twice without reopening or rewinding to the start.

Files are like a tape; as you read the file positions advances until it reaches the end. After that more attempts to read from them only results in no data being returned.

To rewind a file, use the .seek() method:

f.seek(0)

Note that your code seems to do an awful lot of extra work that is entirely not needed. [i for i in ...] merely loops over the input sequence building a copy of the sequence, where no actual copies are needed.

In fact, you don't need to read anything twice, the code can be simplified to:

sample_graph_list_un = []

with open('genes.csv') as f:
    cread = csv.reader(f, delimiter = '\t')
    key_values = [(i, float(j)) for i, j in cread]
    sample_genes = sorted(k for k, v in key_values)
    sample_values = [v for k, v in key_values]  # unsorted for appending first
    sample_graph_list_un.append(sample_values)
    sample_values = sorted(sample_values)       # sorted() creates a copy

Note how the code addends to the sample_graph_list_un list; there is absolutely no need for you to build 6 separately named lists from separate csv files and then later combine them into one list here.

I didn't see how you used the sorted _genes and _values lists, I included them in the code but didn't append them anywhere. Use them in a similar vein, or completely remove the lines with sorted() if you do not need these lists anywhere.

Answer 2

There's a very simple, and very general, solution to this.

Any time you have an iterator (a file, a CSV reader, a generator, whatever) that you want to iterate over multiple times, you can just toss it in a list :

with open('genes.csv') as f:
    cread = list(csv.reader(f, delimiter = '\t'))

Then the rest of your code can remain unchanged (or you can pull it outside of the with statement):

sample_1_dict = {i: float(j) for i, j in cread}
sample_1_list = [x for x in sample_1_dict.items()]
sample_1_genes_sorted = sorted(sample_1_list, key=lambda expvalues: expvalues[0])
sample_1_values_sorted = sorted(sample_1_list, key=lambda expvalues: expvalues[1])
sample_1_genes = [i for i, j in sample_1_values_sorted]
sample_1_values = [j for i, j in sample_1_values_sorted]
sample_1_graph_un = [float(j) for i, j in cread]

The downside of doing this is that you have to build an otherwise-unnecessary list, and you can't start your processing until you've read the whole file. If you can write your whole algorithm as a sequence of one-pass transformations from one iterator to another (eg, generator expressions), that's a huge win.

But in your case, you're already building up a number of lists and dicts, and you can't get to the second one until you've read the whole file, so there's really no cost to building the list at the start.