How do I format real time JSON data (from sensors) into an Excel Workbook in Python?

Question

I am a Python newbie and I want to convert location data from my sensors (which is in JSON) into an Excel Workbook. The sensors are being polled continuously, as such the JSON data is a never ending string.

Now, the interesting part of the code is where the data is encoded in JSON and then "LS: Received" is printed, snippet as under:

 def request(self, jsonStr, timeout = 20):
    try:
        msgToSend = jsonStr.encode(encoding = "utf-8");
        bytesSent = 0
        while bytesSent < len(msgToSend):
            bytesSent += self.svcSocket.send(msgToSend[bytesSent:])

        recvMessage = ""
        while True:
            ready = select.select([self.svcSocket], [], [], timeout)
            if ready[0]:
                recvMessage += self.svcSocket.recv(1024 * 1024).decode(encoding = "utf-8")
                if "\n" in recvMessage: #Complete message ends with \n
                    break
            else:
                #timeout...
                break

and the second snippet is as under:

try:
    while True:
        ready = select.select([lsSocket], [], [], 1)
        if ready[0]:
            recvData = lsSocket.recv(16384)

            if len(recvData) != 0:
                recv = recvData[0:len(recvData)].decode("utf-8")
                print("LS: Received: " + recv)

            else:
                print("LS: Received empty")

        else:
            print("LS: No data, timeout")

The example code I have (straight from the manufacturer) is here ( Python Code for RTLS ). Without proper hardware, it won't compile.

The output of that code is

LS: Received: {"id":"0xDECA303030601EA0","timestamp":15295466,"msgid":69791,"coordinates":{"x":0.664,"y":0.226,"z":0.000,"heading":0.000,"pqf":100},"distances":[{"anchor":"0xDECA3930304142D9","dist":0.854,"dqf":100},{"anchor":"0xDECA333031201D26","dist":1.232,"dqf":100},{"anchor":"0xDECA343032401E27","dist":0.170,"dqf":100}]}

LS: Received: {"id":"0xDECA303030601EA0","timestamp":15295515,"msgid":69792,"coordinates":{"x":0.664,"y":0.226,"z":0.000,"heading":0.000,"pqf":100},"distances":[{"anchor":"0xDECA3930304142D9","dist":0.849,"dqf":100},{"anchor":"0xDECA333031201D26","dist":1.246,"dqf":100},{"anchor":"0xDECA343032401E27","dist":0.170,"dqf":100}]}

LS: Received: {"id":"0xDECA303030601EA0","timestamp":15295565,"msgid":69793,"coordinates":{"x":0.664,"y":0.226,"z":0.000,"heading":0.000,"pqf":100},"distances":[{"anchor":"0xDECA3930304142D9","dist":0.830,"dqf":100},{"anchor":"0xDECA333031201D26","dist":1.222,"dqf":100},{"anchor":"0xDECA343032401E27","dist":0.170,"dqf":100}]}

This keeps on going indefinitely until I close the program.

The first id is that of a mobile "tag" which moves in an area encapsulated by "anchors." The ids are constant, they never change. Of course, with more anchors, there will be more but their 'name' is always the same .

The coordinates of the tag are in the x,y,z direction. They change constantly .

Dist is the distance of the anchor from the tag. It changes constantly .

Msgid is the message id, it increments with each message.

Timestamp also changes .

The rest of the data can be ignored.

The spreadsheet I have in mind should just display the x,y,z and dist as changing parameters, as follows (can't post images yet):

Sample Excel Sheet

I have tried to use xlwt package, but I can't, for the life of me, figure out how to parse this incoming JSON data. I tried to use split() but it didn't work. My inexperience with Python is really slowing me down.

Any advice, however insignificant, will be highly appreciated. Many thanks.

Answer 1

Never mind, you're not doing what I think you're doing. Try using the simplejson module. Here's how you might do it:

>>> import simplejson
>>> json = simplejson.loads(recvData.decode('utf-8'))

At this point, json should be a dictionary representing that response.

I think I would also create CSVs rather than Excel documents. XLWT is easy enough once you get used to it, but you can create CSV files much more easily, and you aren't doing any fancy formatting.

Answer 2

You may try using my library pyexcel to help you export your python data into any type of excel files(csv, xls, xlsx, or ods).

Here is a piece of example code:

>>> import pyexcel as pe
>>> import pyexcel.ext.xls
>>> data=[[1,2]]
>>> pe.save_as(array=data, out_file='test.xls')