要将CSV文件读入字典？

Question

First I want to start off by saying I am NOT asking you to write code. I only want to discuss and get feed back on what would be the best way to go about writing this program because I am stuck on figuring out how to break down the problem.

My program is supposed to open a CSV file which contains 7 columns:

Name of the state,Crop,Crop title,Variety,Year,Unit,Value. 

Here is part of the file:

Indiana,Corn,Genetically engineered (GE) corn,Stacked gene varieties,2012,Percent of all corn planted,60
Indiana,Corn,Genetically engineered (GE) corn,Stacked gene varieties,2013,Percent of all corn planted,73
Indiana,Corn,Genetically engineered (GE) corn,Stacked gene varieties,2014,Percent of all corn planted,78
Indiana,Corn,Genetically engineered (GE) corn,Stacked gene varieties,2015,Percent of all corn planted,76
Indiana,Corn,Genetically engineered (GE) corn,Stacked gene varieties,2016,Percent of all corn planted,75
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2000,Percent of all corn planted,11
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2001,Percent of all corn planted,12
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2002,Percent of all corn planted,13
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2003,Percent of all corn planted,16
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2004,Percent of all corn planted,21
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2005,Percent of all corn planted,26
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2006,Percent of all corn planted,40
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2007,Percent of all corn planted,59
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2008,Percent of all corn planted,78
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2009,Percent of all corn planted,79
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2010,Percent of all corn planted,83
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2011,Percent of all corn planted,85
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2012,Percent of all corn planted,84
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2013,Percent of all corn planted,85
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2014,Percent of all corn planted,88
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2015,Percent of all corn planted,88
Indiana,Corn,Genetically engineered (GE) corn,All GE varieties,2016,Percent of all corn planted,86

Then read each line into a dictionary. There are many many lines in this text file, the only lines I want/need are the lines whose Variety column reads "All GE varieties." Please note each state also has multiple lines. The next step is to use a user input of a crop and only examine the data for that crop. The final step is to then figure out (for each state) what is the max and min value and its corresponding year and print it.

The way I was thinking of going about this was possibly creating a set for each line, checking if "All GE varieties" was in the set and if it is then add that to a dictionary. And then do something similar for the crop?

My biggest dilemma is probably that 1.) I don't know how to go about ignoring lines that don't contain "All GE varieties." Do I do that before or after I create the dictionary? and 2.) I know how to create a dictionary with one value and one key, but how would I go about adding the rest of the values to the key? Do you do that with sets? or lists?

Answer 1

Figuring out if "All GE varieties" is in the string is relatively straightforward - use the in keyword:

with open(datafile, 'r') as infile:
    for line in file:
        if "All GE varieties" in line:
            # put into line into data structure

For the data structure, I'm partial to lists of dictionaries, where each dictionary has a defined set of keys:

myList = [ {}, {}, {}, ... ]

The problem in this case is I'm not sure what you would use as the key, if each field is a value. Also remember the split() command can help:

varieties = []
with open(datafile, 'r') as infile:
    for line in file:
        if "All GE varieties" in line:
            varieties.append(line.split(','))

This would give you a list (varieties) containing lists, each of which single fields from each line.

Something like this:

varieties = [['Indiana','Corn','Genetically engineered (GE) corn','All GE varieties','2000','Percent of all corn planted','11'], ['Indiana','Corn','Genetically engineered (GE) corn','All GE varieties','2001','Percent of all corn planted','12'], ... ]

From here it would be fairly easy to pick out the state or year, etc. using slices (2D array).

Answer 2

As previously mentioned, you can use the csv module to read in the csv file. I wasn't exactly sure how you wanted the data structured after the state key but I thought it might be nicer to be able to look up each particular crop_title and then be able to access the value for each year separately.

In[33]: from collections import defaultdict
   ...: from csv import reader
   ...: 
   ...: crops = defaultdict(lambda: defaultdict(dict))
   ...: with open('hmm.csv', 'r') as csvfile:
   ...:     cropreader = reader(csvfile)
   ...:     for row in cropreader:
   ...:         state, crop_type, crop_title, variety, year, unit, value = row
   ...:         if variety == 'All GE varieties':
   ...:             crops[state][crop_title][year] = value
   ...: 
In[34]: crops
Out[34]: 
defaultdict(<function __main__.<lambda>>,
            {'Indiana': defaultdict(dict,
                         {'Genetically engineered (GE) corn': {'2000': '11',
                           '2001': '12',
                           '2002': '13',
                           '2003': '16',
                           '2004': '21',
                           '2005': '26',
                           '2006': '40',
                           '2007': '59',
                           '2008': '78',
                           '2009': '79',
                           '2010': '83',
                           '2011': '85',
                           '2012': '84',
                           '2013': '85',
                           '2014': '88',
                           '2015': '88',
                           '2016': '86'}})})
In[35]: crops['Indiana']['Genetically engineered (GE) corn']['2000']
Out[35]: '11'
In[36]: crops['Indiana']['Genetically engineered (GE) corn']['2015']
Out[36]: '88'

You could also convert year and value into integers like this crops[state][crop_title][int(year)] = int(value) which would allow you to make calls like this (where the return value is an integer):

In[38]: crops['Indiana']['Genetically engineered (GE) corn'][2015]
Out[38]: 88

Answer 3

I put your data into a file named "crop_data.csv". Here's some code that uses the standard csv module to read each line into its own dictionary. We use a simple if test to make sure we only keep lines where 'Variety' == 'All GE varieties' , and we store the data for each state in all_data , which is a dictionary of lists, one list per state. Since the state 'Name' is used as the key in all_data we don't need to keep it in the row dict, similarly we can discard the 'Variety', since we don't need that info anymore.

After all the data is gathered we can print it nicely using the json module.

Then we loop over all_data , state by state, and calculate its maximum and minimum.

import csv
from collections import defaultdict
import json

filename = 'crop_data.csv'

fieldnames = 'Name,Crop,Title,Variety,Year,Unit,Value'.split(',')

all_data = defaultdict(list)

with open(filename) as csvfile:
    reader = csv.DictReader(csvfile, fieldnames=fieldnames)
    for row in reader:
        # We only want 'All GE varieties'
        if row['Variety'] == 'All GE varieties':
            state = row['Name']
            # Get rid of unneeded fields
            del row['Name'], row['Variety']
            # Store it as a plain dict
            all_data[state].append(dict(row))

# Show all the data
print(json.dumps(all_data, indent=4))

#Find minimums & maximums

# Extract the 'Value' field from dict d and convert it to a number
def value_key(d):
    return int(d['Value'])

for state, data in all_data.items():
    print(state)
    row = min(data, key=value_key)
    print('min', row['Value'], row['Year'])

    row = max(data, key=value_key)
    print('max', row['Value'], row['Year'])

output

{
    "Indiana": [
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2000",
            "Unit": "Percent of all corn planted",
            "Value": "11"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2001",
            "Unit": "Percent of all corn planted",
            "Value": "12"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2002",
            "Unit": "Percent of all corn planted",
            "Value": "13"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2003",
            "Unit": "Percent of all corn planted",
            "Value": "16"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2004",
            "Unit": "Percent of all corn planted",
            "Value": "21"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2005",
            "Unit": "Percent of all corn planted",
            "Value": "26"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2006",
            "Unit": "Percent of all corn planted",
            "Value": "40"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2007",
            "Unit": "Percent of all corn planted",
            "Value": "59"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2008",
            "Unit": "Percent of all corn planted",
            "Value": "78"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2009",
            "Unit": "Percent of all corn planted",
            "Value": "79"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2010",
            "Unit": "Percent of all corn planted",
            "Value": "83"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2011",
            "Unit": "Percent of all corn planted",
            "Value": "85"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2012",
            "Unit": "Percent of all corn planted",
            "Value": "84"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2013",
            "Unit": "Percent of all corn planted",
            "Value": "85"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2014",
            "Unit": "Percent of all corn planted",
            "Value": "88"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2015",
            "Unit": "Percent of all corn planted",
            "Value": "88"
        },
        {
            "Crop": "Corn",
            "Title": "Genetically engineered (GE) corn",
            "Year": "2016",
            "Unit": "Percent of all corn planted",
            "Value": "86"
        }
    ]
}
Indiana
min 11 2000
max 88 2014

---

Note that in this data there are 2 years with the value of 88. You could use a fancier key function than value_key if you want to break ties by year. Or you can use value_key to sort the whole state data list, so you can easily extract all the lowest and highest records. Eg, in that for state, data loop do

data.sort(key=value_key)
print(json.dumps(data, indent=4))

and it will print all the records for that state in numerical order.