Python将txt文件转换为JSON剥离和元素排序

Question

I have a txt file that contains data split by spaces such as:

2017-05-16 00:44:36.151724381 +43.8187 -104.7669 -004.4 00.6 00.2 00.2 090 C
2017-05-16 00:44:36.246672534 +41.6321 -104.7834 +004.3 00.6 00.3 00.2 130 C
2017-05-16 00:44:36.356132768 +46.4559 -104.5989 -004.2 01.1 00.4 00.2 034 C

and I would like to convert it into JSON data to something like:

{"dataset": "Lightning","observation_date": "20170516004436151", "location": { "type": "point", "coordinates": [43.8187, -104.7669]}}
{"dataset": "Lightning","observation_date": "20170516004436246", "location": { "type": "point", "coordinates": [41.6321, -104.7834]}}
{"dataset": "Lightning","observation_date": "20170516004436356", "location": { "type": "point", "coordinates": [46.4559, -104.5989]}}

where I have to append a 'dataset':'lightning' key/val pair, combine and strip the date and time, and combine the lat/lng into a dict before doing any json conversion.

But right now I still get the date and time elements without being stripped of the "-" and ":" characters like:

{"observation_date": "2017-05-1600:44:36.151724381", "location": {"type": "point", "coordinates": ["+43.8187", "-104.7669"]}, "dataset": "Lightning"}
{"observation_date": "2017-05-1600:44:36.246672534", "location": {"type": "point", "coordinates": ["+41.6321", "-104.7834"]}, "dataset": "Lightning"}
{"observation_date": "2017-05-1600:44:36.356132768", "location": {"type": "point", "coordinates": ["+46.4559", "-104.5989"]}, "dataset": "Lightning"}

What I coded so far:

import json
import sys
def convert(filename):
    dataDict = {}
    txtFile = filename[0]
    print "Opening TXT file: ",txtFile
    infile = open(txtFile, "r")
    for line in infile:
        lineStrip = line.strip()
        parts = [p.strip() for p in lineStrip.split()]
        date = parts[0].strip("-") #trying to get rid of "-" but not working
        time = parts[1].strip(":") #trying to get rid of ":" and "." but not working
        dataDict.update({"dataset":"Lightning"})
        dataDict.update({"observation_date": date + time})
        dataDict.update({"location": {"type":"point", "coordinates": [parts[2], parts[3]]}})
        json_filename = txtFile.split(".")[0]+".json"
        jsonf = open(json_filename,'a')
        data = json.dumps(dataDict)
        jsonf.write(data + "\n")
        print dataDict
    infile.close()
    jsonf.close()   
if __name__=="__main__":
    convert(sys.argv[1:])

But I'm not sure how to strip the "-", ".", and ":" as well as place the "dataset":"lightning" element in the front.

Answer 1

This should work

date = parts[0].replace("-",'') #trying to get rid of "-" but not working

time = parts[1].replace(":",'') #trying to get rid of ":" and "." but not working

Answer 2

You should do:

date = parts[0].replace('-', '') time = parts[1].replace(':' '')

To get the dataset up front in JSON, the only option you have is sort the keys:

data = json.dumps(dataDict, sort_keys=True)

You should also consider doing

dataDict["dataset"] = "Lightning"

instead of the .update .

Answer 3

Python dictionaries are unordered, so you can't specify the "dataset":"lightning" element to be first. For that I would use an OrderedDict instead or sort the json as others have mentioned.

In order to format the time correctly, I'd use a datetime object as such:

import datetime

date_string = parts[0] + parts[1]
format = "%Y-%d-%m%H:%M:%S.%f"
dt = datetime.strptime(date_string, format)
new_date_string = dt.strftime("%Y%d%m%H%M%S")

Using a datetime object is helpful because it plays nicely with pandas and numpy if you continue to work on the data beyond spitting out the json. It also supports mathematical operations and time zone localization if you need it to.