Reading in CSV file to add data to a dictionary or tuple

Question

I am new to Python. I need to rewrite my stock picking program and need help getting started.

The first module is a master data creating program. It reads a CSV EOD (end of day) stock data file.

Then it parses through a master file adding that days stock information to an array (of 35 elements).

Each array has 5 columns containing the EOD information.

The original program was indexed on stock name. Indexing is not important for the master file is only accessed sequentially in the second main stock program. What is needed is that the stock name to be used as a header for the (arrayed) records in the master file.

What is needed is a way to open and read the CSV file. Comparing each stock name if stka == stkb update the information in main.idx .

Pseudo code:

    Open-r CVS
    Open-rwb main
    While (cvs)
            Readline(CVS)
            Readline (main)
      If cvs == main
            Add to record
            Wright (main)
      else
      If cvs > main
            Readline (main)
      else
      If cvs < main
            Readline (cvs)

Here is some code that handles a CSV program and my attempt, found under Updating CSV files automatically .

My needs are simpler in that I only need to read in the CSV file. The master file can be in any Python data set list , dictionary , or tuple .

Thanks for any helping in pointing me in the right direction.

Answer 1

You aren't very clear about the file formats you are using, or what you need to do with the records once you read them in. But taking some file formats from your other question on reading CSV files , I'll give you an example program to work from.

Here is a master file m.csv.txt . It's a Comma Separated Values (CSV) file, and the first field is an identifier.

AAC,D,20111207,9.83,9.83,9.83,9.83,100
AACC,D,20111207,3.46,3.47,3.4,3.4,13400
AACOW,D,20111207,0.3,0.3,0.3,0.3,500
AAME,D,20111207,1.99,1.99,1.95,1.99,8600
AAON,D,20111207,21.62,21.9,21.32,21.49,93200
AAPL,D,20111207,389.93,390.94,386.76,389.09,10892800

Here's a daily update file d.csv.txt . It has the same format. Again, the first field is an identifier. If the identifier is the same as an identifier from the master file, the entries should be combined. Otherwise, new identifiers in the daily file get added to the overall list.

AACC,D,20120127,4.01,4.02,4.03,4.04,40000
B,D,20120127,4.01,4.02,4.03,4.04,40000

Here is a simple program to read in the master file, using the csv module , and put an entry for each line into a dictionary stocks . The first field is used as the key for the dictionary. This is a very simple data structure, but you said the master file can be in any data format. Then we print out the stocks dictionary.

import csv
mainReader = csv.DictReader( open("m.csv.txt","rb"), ["id"],"others")

newReader = csv.DictReader( open("d.csv.txt","rb"), ["id"],"others")
stocks = {}
for line in mainReader:
    stocks[line['id']] = line

print stocks   # output hand-formatted
{
   'AAC':  {'id':'AAC', 
            'others': ['D','20111207','9.83','9.83','9.83','9.83','100']},
   'AACC': {'id':'AACC', 
            'others': ['D', '20111207', '3.46', '3.47', '3.4', '3.4', '13400']},
   'AAME': {'id': 'AAME',
            'others': ['D', '20111207', '1.99', '1.99', '1.95', '1.99', '8600']},
   'AACOW': {'id': 'AACOW',
            'others': ['D', '20111207', '0.3', '0.3', '0.3', '0.3', '500']},
   'AAPL': {'id': 'AAPL', 'others': 
            ['D','20111207','389.93','390.94','386.76','389.09','10892800']},
   'AAON': {'id': 'AAON', 
            'others': ['D', '20111207', '21.62', '21.9', '21.32', '21.49', '93200']}
}

Now the program goes on to read in the daily update file, again using the csv module. You aren't clear how an entry should be combined with the entry from the main file if they have the same identifier. For this example, I'll just let the entry from the daily file completely replace the existing entry. They we print out the resulting stocks .

for line in newReader:
    # can use if line['id'] in stocks: to check for an existing record if desired
    # in this example, we blindly overwrite any existing entry
    stocks[line['id']] = line

print stocks    # output hand-formatted
{
    'AAME': {'id': 'AAME', 
             'others': ['D', '20111207', '1.99', '1.99', '1.95', '1.99', '8600']},
    'B': {'id': 'B', 
          'others': ['D', '20120127', '4.01', '4.02', '4.03', '4.04', '40000']},
    'AACC': {'id': 'AACC',
             'others': ['D', '20120127', '4.01', '4.02', '4.03', '4.04', '40000']},
    'AAPL': {'id': 'AAPL',
        'others': ['D','20111207','389.93','390.94','386.76','389.09','10892800']},
    'AAON': {'id': 'AAON', 
        'others': ['D', '20111207', '21.62', '21.9', '21.32', '21.49', '93200']},
    'AACOW': {'id': 'AACOW', 
        'others': ['D', '20111207', '0.3', '0.3', '0.3', '0.3', '500']}
}

Notice that the dictionary data structure doesn't have any particular order when printed. You want sorted output? Use the items() method for dictionaries to get a list, and .sort() to sort the list. (There are many other ways to do this.)

s = stocks.items()
s.sort()
print s   # output hand-formatted
[
    'AACC': {'id': 'AACC',
             'others': ['D', '20120127', '4.01', '4.02', '4.03', '4.04', '40000']},
    'AACOW': {'id': 'AACOW', 
        'others': ['D', '20111207', '0.3', '0.3', '0.3', '0.3', '500']},
    'AAME': {'id': 'AAME', 
             'others': ['D', '20111207', '1.99', '1.99', '1.95', '1.99', '8600']},
    'AAON': {'id': 'AAON', 
        'others': ['D', '20111207', '21.62', '21.9', '21.32', '21.49', '93200']},
    'AAPL': {'id': 'AAPL',
        'others': ['D','20111207','389.93','390.94','386.76','389.09','10892800']},
    'B': {'id': 'B', 
          'others': ['D', '20120127', '4.01', '4.02', '4.03', '4.04', '40000']}
]

Do you want to write out an updated master file. Use csv.DictWriter() or csv.writer() . You can do this line-by-line, or output a complete in-memory list like stocks all at once.