Parsing text file into csv

Question

I have a following sample data(it's just a portion)from my text file. I am trying to extract three keys including timestamp,dataFrame, and rssi into csv file.

packet"{\"test\":{\"id\":1479238177559,\"deveui\":\"0000000033035032\",\"timestamp\":\"2016-11-15T19:29:37.559Z\",\"dataFrame\":\"ABzuPdVNxrSEAV8=\",\"fcnt\":81,\"port\":5,\"rssi\":6,\"snr\":9.5,\"sf_used\":10,\"cr_used\":\"4/5\",\"device_redundancy\":0,\"time_on_air_ms\":288.76800000000003,\"decrypted\":true}}"
Received message in at 2016-11-15 14:29:43.611000
packet"{\"test\":{\"id\":1479238184069,\"deveui\":\"0000000033035032\",\"timestamp\":\"2016-11-15T19:29:44.069Z\",\"dataFrame\":\"ABzuPdVNxrSEAV8=\",\"fcnt\":82,\"port\":5,\"rssi\":6,\"snr\":8.5,\"sf_used\":10,\"cr_used\":\"4/5\",\"device_redundancy\":0,\"time_on_air_ms\":288.76800000000003,\"decrypted\":true}}"
Received message in at 2016-11-15 14:29:49.225000
packet"{\"test\":{\"id\":1479238189685,\"deveui\":\"0000000033035032\",\"timestamp\":\"2016-11-15T19:29:49.685Z\",\"dataFrame\":\"ABzuPdVNxrSEAV8=\",\"fcnt\":83,\"port\":5,\"rssi\":7,\"snr\":9.5,\"sf_used\":10,\"cr_used\":\"4/5\",\"device_redundancy\":0,\"time_on_air_ms\":288.76800000000003,\"decrypted\":true}}"
Received message in at 2016-11-15 14:29:56.410000
packet"{\"testl\":{\"id\":1479238196868,\"deveui\":\"0000000033035032\",\"timestamp\":\"2016-11-15T19:29:56.868Z\",\"dataFrame\":\"ABzuPdVNxrSEAV8=\",\"fcnt\":84,\"port\":5,\"rssi\":3,\"snr\":9.8,\"sf_used\":10,\"cr_used\":\"4/5\",\"device_redundancy\":0,\"time_on_air_ms\":288.76800000000003,\"decrypted\":true}}"

Answer 1

This is apparently data that has accidentally been JSON encoded twice, so it can be decoded twice to get a nice dictionary:

import json

with open('log.txt') as infile:
    packet = []
    for line in infile:
        if line.startswith('packet"{'):
            # Remove 'packet' prefix
            line = line[len('packet'):]
            packet = json.loads(json.loads(line))
            print('Packet:')
            print(packet)
            packet = packet.values()[0]
            print('Values:')
            print(packet['timestamp'], packet['dataFrame'], packet['rssi'])

Output:

Packet:
{u'test': {u'decrypted': True, u'fcnt': 81, u'timestamp': u'2016-11-15T19:29:37.559Z', u'dataFrame': u'ABzuPdVNxrSEAV8=', u'id': 1479238177559, u'sf_used': 10, u'snr': 9.5, u'cr_used': u'4/5', u'deveui': u'0000000033035032', u'device_redundancy': 0, u'rssi': 6, u'port': 5, u'time_on_air_ms': 288.76800000000003}}
Values:
(u'2016-11-15T19:29:37.559Z', u'ABzuPdVNxrSEAV8=', 6)
Packet:
{u'test': {u'decrypted': True, u'fcnt': 82, u'timestamp': u'2016-11-15T19:29:44.069Z', u'dataFrame': u'ABzuPdVNxrSEAV8=', u'id': 1479238184069, u'sf_used': 10, u'snr': 8.5, u'cr_used': u'4/5', u'deveui': u'0000000033035032', u'device_redundancy': 0, u'rssi': 6, u'port': 5, u'time_on_air_ms': 288.76800000000003}}
Values:
(u'2016-11-15T19:29:44.069Z', u'ABzuPdVNxrSEAV8=', 6)
Packet:
{u'test': {u'decrypted': True, u'fcnt': 83, u'timestamp': u'2016-11-15T19:29:49.685Z', u'dataFrame': u'ABzuPdVNxrSEAV8=', u'id': 1479238189685, u'sf_used': 10, u'snr': 9.5, u'cr_used': u'4/5', u'deveui': u'0000000033035032', u'device_redundancy': 0, u'rssi': 7, u'port': 5, u'time_on_air_ms': 288.76800000000003}}
Values:
(u'2016-11-15T19:29:49.685Z', u'ABzuPdVNxrSEAV8=', 7)
Packet:
{u'testl': {u'decrypted': True, u'fcnt': 84, u'timestamp': u'2016-11-15T19:29:56.868Z', u'dataFrame': u'ABzuPdVNxrSEAV8=', u'id': 1479238196868, u'sf_used': 10, u'snr': 9.8, u'cr_used': u'4/5', u'deveui': u'0000000033035032', u'device_redundancy': 0, u'rssi': 3, u'port': 5, u'time_on_air_ms': 288.76800000000003}}
Values:
(u'2016-11-15T19:29:56.868Z', u'ABzuPdVNxrSEAV8=', 3)

Answer 2

You can try this approach:

import pandas as pd

content = []
for c in open('log.txt').readlines():
    if c.startswith('packet"{'):
        content.append(c[7:-2].decode('string_escape'))

df = pd.concat([pd.read_json(line, orient='index') for line in content])

df[['dataFrame', 'rssi', 'timestamp']].to_csv('out.csv', index=False, header=None)

Data in the out.csv file is:

ABzuPdVNxrSEAV8=,6,2016-11-15 19:29:37.559
ABzuPdVNxrSEAV8=,6,2016-11-15 19:29:44.069
ABzuPdVNxrSEAV8=,7,2016-11-15 19:29:49.685
ABzuPdVNxrSEAV8=,3,2016-11-15 19:29:56.868