How to append new row to csv file with pandas?

Question

I want to save data that I received from mosquitto broker as csv file. Below are the script for mqtt_subscribe.py

import paho.mqtt.client as mqtt
import pandas as pd

def on_connect(client, userdata, rc):
    print("Connected with result code "+str(rc))
    client.subscribe("test")

def on_message(client, userdata, msg):
    print str(msg.payload)
    datas = map(int, msg.payload)
    df = pd.DataFrame(data=datas, columns=['numbers'])

    f = open("test.csv", 'a')
    df.to_csv(f)
    f.close()

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("localhost", 1883, 60)

client.loop_forever()

This script will print random number/data received from the broker (until I manually stop the script) like this(Example)

Connected with result code 0
4
7
7

I am hoping to write this result in CSV file like this

,numbers
0,4
1,7
2,7

but instead I got this

,numbers
0,4
,numbers
0,7
,numbers
0,7

Am I missing something here? I believe it is because method on_message keep overwriting the dataframe with the column but I do not know where should I initialize the dataframe other than inside on_message method.

Thank you in advance.

Answer 1

You can use something like this for appending panda dataframe df to test.csv without including headers:

with open('test.csv', 'a') as csv_file:
    df.to_csv(csv_file, header=False)

The usage of 'with open' helps us to not bother about explicitly closing file