Read and process a text file and save to csv

Question

The files I have seem to be in a "dict" format...

file header is as follows: time,open,high,low,close,volume

next line is as follows: {"t":[1494257340],"o":[206.7],"h":[209.3],"l":[204.50002],"c":[204.90001],"v":[49700650]}`

    import csv
    with open ('test_data.txt', 'rb') as f:

    for line in f:
        dict_file = eval(f.read())
        time = (dict_file['t'])    # print (time) result [1494257340]
        open_price = (dict_file['o'])    # print (open_price) result [206.7]
        high = (dict_file['h'])    # print (high) result [209.3]
        low = (dict_file['l'])    # print (low) result [204.50002]
        close = (dict_file['c'])    # print (close) result [204.90001]
        volume = (dict_file['v'])    # print (volume) result [49700650]

        print (time, open_price, high, low, close, value)

# print result [1494257340] [206.7] [209.3] [204.50002] [204.90001] [49700650]

# I need to remove the [] from the output.

# expected result 

# 1494257340, 206.7, 209.3, 204.50002, 204.90001, 49700650

the result I need is (change time ("epoch date format") to dd,mm,yy

5/8/17, 206.7, 209.3, 204.50002, 204.90001, 49700650

so I know I need the csv.writer function

Answer 1

I see a number of problems in the code you submitted. I recommend you to break your task into small pieces and see if you can make them work individually. So what are you trying to do is:

1. open a file
2. read the file line by line
3. eval each line to get a dict object
4. get values from that object
5. write those values in a (separate?) csv file

Right?

Now do each one, one small step at the time

1. opening a file.

You're pretty much on point there:

with open('test_data.txt', 'rb') as f:
    print(f.read())

# b'{"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}\n'

You can open the file in r mode instead, it will give you strings instead of byte type objects

with open('test_data.txt', 'r') as f:
    print(f.read())

# {"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}

It might cause some problems but should work since eval can handle it just fine (at least in python 3)

2. read the file line by line

with open('test_data.txt', 'rb') as f:
    for line in f:
        print(line)

# b'{"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}\n'

Here is another problem in your code, you're not using line variable and trying to f.read() instead. This will just read entire file (starting from the second line, since the first one is been read already). Try to swap one for another and see what happens

3. eval each line to get a dict object

Again. This works fine. but I would add some protection here. What if you get an empty line in the file or a misformatted one. Also if this file comes from an untrusted source you may become a victim of a code injection here, like if a line in your file changed to:

print("You've been hacked") or {"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}

with open('test_data.txt', 'rb') as f:
    for line in f:
        dict_file = eval(line)
        print(dict_file)

# You've been hacked
# {'t': [1494257340], 'o': [207.75], 'h': [209.8], 'l': [205.75], 'c': [206.35], 'v': [61035956]}

I don't know your exact specifications, but you should be safer with json.loads instead.

...

---

Can you continue on your own from there?

4. get values from the object

I think dict_file['t'] doesn't give you the value you expect.

What does it give you?

Why?

How to fix it?

5. write those values in a csv file

Can you write some random string to a file?

What scv format looks like? Can you format your values to match it

Check the docs for csv module, can it be of help to you?

And so on and so forth...

---

EDIT: Solution

# you can save the print output in a file by running:
# $ python convert_to_csv.py > output.cvs
import datetime, decimal, json, os


CSV_HEADER = 'time,open,high,low,close,volume'


with open('test_data.txt', 'rb') as f:

    print(CSV_HEADER)

    for line in f:
        data = json.loads(line, parse_float=decimal.Decimal)
        data['t'][0] = datetime.datetime.fromtimestamp(data['t'][0]) \
            .strftime('%#d/%#m/%y' if os.name == 'nt' else '%-d/%-m/%y')
        print(','.join(str(data[k][0]) for k in 'tohlcv'))

Running:

$ cat test_data.txt
{"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}
{"t":[1490123123],"o":[107.75],"h":[109.8],"l":[105.75],"c":[106.35],"v":[11035956]}
{"t":[1491234234],"o":[307.75],"h":[309.8],"l":[305.75],"c":[306.35],"v":[31035956]}

$ python convert_to_csv.py
time,open,high,low,close,volume
8/5/17,207.75,209.8,205.75,206.35,61035956
21/3/17,107.75,109.8,105.75,106.35,11035956
3/4/17,307.75,309.8,305.75,306.35,31035956