Intermittent syntax error in a bash script involving echo, a python script, grep, sed, bc, and date

Question

I have written a python script to take magnetic field measurements from a Raspberry Pi Astro "Sense Hat" sensor. It is called "mag-AstroPi.py":

#!/usr/bin/python
from sense_hat import SenseHat
sense = SenseHat()
raw = sense.get_compass_raw()
#print(x: {x}, y: {y}, z: {z}.format(**raw))
#alternatives
print(sense.compass_raw)

This is the script provided by element14, the manufacturer of the Sense Hat.

The script outputs magnetic field data in three axes (X, Y, and Z) in microteslas, along with a bunch of extra characters:

pi@raspberrypi ~ $ python mag-AstroPi.py

{'y': 13.895279884338379, 'x': -1.1642401218414307, 'z': -0.4132799804210663}

I need to remove the extra characters, multiply the values by 1,000 in order to convert them into nanoteslas (standard SI unit for my particular application), and then log the multiplied value alongside the date and time into a file. This needs to happen every two seconds.

I want there to be three separate log files - one for the X axis, one for the Y axis, and one for the Z axis. However, for now, I am just working with the Y-axis data. Once I get the Y-axis data logging working, I can then duplicate and alter for the two other axes.

So I wrote a bash script, AstroPiMagLogger.sh, which runs at boot via a cron job:

#!/bin/bash
while true
do
        echo $(python mag-AstroPi.py -n | grep "y" | cut -d " " -f2 | cut -c 1-18 | sed 's/$/*1000/' | bc; date +"%Y,%m,%d,%T,%Z") >> rawysecnT.txt
        sleep 2
done

This should extract the Y-axis value only, multiply it by 1,000, and then save it alongside the current date, time, and time zone into a new text file, rawysecnT.txt.

It works, sorta... here are the contents of rawysecnT.txt:

13703.761100769043000 2015,09,14,08:56:41,UTC

13703.761100769043000 2015,09,14,08:56:44,UTC

13613.041877746582000 2015,09,14,08:56:46,UTC

13794.480323791504000 2015,09,14,08:56:49,UTC

13804.560661315918000 2015,09,14,08:56:52,UTC

13875.120162963867000 2015,09,14,08:56:55,UTC

13633.201599121094000 2015,09,14,08:56:58,UTC

2015,09,14,08:57:00,UTC

2015,09,14,08:57:03,UTC

13744.080543518066000 2015,09,14,08:57:06,UTC

14016.241073608398000 2015,09,14,08:57:09,UTC

As you can see, it works most of the time. But every now and then, it doesn't log the magnetic field measurement to the file; it only logs the date and time.

Earlier today, I had the logging working perfectly, but that was before I added the code to multiply the magnetic data by 1000 (ie earlier today, the script was only logging the original magnetic data in microteslas, along with the date/time). I have several hours worth of data like that without any errors at all, so it's apparent I've stuffed something up when adding in the code for multiplication of the magnetic measurement.

I decided to run the following directly in the command line (rather than through the script), in order to debug.

echo $(python mag-AstroPi.py -n |grep "y" | cut -d " " -f2 | cut -c 1-18 | sed 's/$/*1000/' | bc; date +"%Y,%m,%d,%T,%Z")

Predictably, this worked about a dozen times, with the following output printed to the terminal, which is exactly how I want it:

14167.440414428711000 2015,09,14,09:07:30,UTC

and then, one last time, it returned the following error:

(standard_in) 1: syntax error

2015,09,14,09:07:59,UTC

Given that the error is intermittent, and I'm fairly new to programming (I've only been at it about a month), I've got no idea what could possibly be the issue.

I would appreciate any thoughts anyone may have as to why this is working most of the time but not all of the time.

---

The two sample outputs requested in the comments are as follows:

pi@raspberrypi ~ $ python mag-AstroPi.py -n | grep "y" | cut -d " " -f2 | cut -c 1-18 | sed 's/$/*1000/' | bc; date +"%Y,%m,%d,%T,%Z" 
14076.720237731934000 2015,09,14,09:53:33,UTC 

pi@raspberrypi ~ $ python mag-AstroPi.py -n 
{'y': 13.935601234436035, 'x': -1.506960153579712, 'z': 0.24192002415657043}

Answer 1

It looks like what you're trying to do could (and almost certainly should) be done within the Python script itself. get_compass_raw() is returning you a dictionary, so you can extract the y value (and multiply by 1000) directly:

raw = sense.get_compass_raw()
y_component_nT = raw['y'] * 1000

To add your timestamp, I'd use the built-in datetime module:

from datetime import datetime
now = datetime.now()

You can then format the time however you want, using now.strftime(format) , where format is a format string built up as shown in the docs .

I'll leave the challenges of writing to a file in python and pausing execution to you - they're already covered in many good answers on this site and elsewhere.

Answer 2

I agree that this is better done in the Python script itself, and if you're unfamiliar with Python, this is a good opportunity to learn some small parts of it. As for your pipeline, I think the key issue is in cut -c 1-18 | sed 's/$/*1000/' | bc cut -c 1-18 | sed 's/$/*1000/' | bc cut -c 1-18 | sed 's/$/*1000/' | bc . The floating point numbers are not guaranteed to be a specific width or even format since you just printed a dictionary without requesting any formatting, so sometimes this will include the comma (or final brace for the last component), or be in scientific notation such as 2.34e-07 . bc does not understand those forms. Also, as the script only prints the one line with all values, grep does nothing.

If I were to use a pipeline like this to extract a value, I would probably use something like sed -e "s/.*'y': \\([-Ee.0-9]*\\).*/\\1/" -e "s/[Ee]\\(.*\\)/*10^\\1/" instead of the cuts (the latter substitution converts e forms to bc compatible expressions). On top of that, bc has some specific rules regarding precision, which mean the exponent handling requires you to set scale or everything becomes zero.

Answer 3

Obviously I can't test this code properly without a Sense Hat sensor, but you should find it helpful. It does an infinite loop, so you need to send it a Ctrl C to kill it.

#!/usr/bin/env python

import time
from sense_hat import SenseHat

sense = SenseHat()

#Un-comment the following line to ensure the compass is on and the gyro & accelerometer are off
#sense.set_imu_config(True, False, False)

while True:
    raw_field = sense.get_compass_raw()
    now = time.strftime("%Y,%m,%d,%X,%Z")
    print(raw_field['y'] * 1000, now)
    time.sleep(2)

If you're using Python 2 you'll need to remove the parentheses in the print line, or add from __future__ import print_function before the other import lines.

It's not hard to adapt this code to write directly to a named file rather than having to pipe its output. But I'll leave that as an Exercise for the Reader. :) And it's also easy to modify it write to 3 files for x, y, and z.

---

I suspect that sometimes the SenseHat simply doesn't return the expected magnetic field data, but it's hard to tell from the info you posted what it does return when it does that. It possibly returns a dictionary containing empty data values, or it could return a totally empty dictionary.

So if you still get bad output, let me know what it looks like & any associated error message, and I should be able to show you how to fix it.