Insert data to MySQL table every second (one time per second)

Question

i'm new in Python, Raspberry Pi and MySQL and i hope that you can help me. I'm trying to write a script in Python that could insert data every second into a MySQL table. I can insert data but is not periodical like i want, I already tried a lot and i can't find the solution to my problem. Here goes my Python code and the data inserted into the MySQL table:

Python code:

#!/usr/bin/env python

import MySQLdb
import time

while True:
    db = MySQLdb.connect("localhost", "mauro", "12345", "temps")
    curs=db.cursor()
    try:
        curs.execute ("""INSERT INTO thetemps 
                values(0, CURRENT_DATE(), NOW(), 28)""")
        db.commit()
        print "Data committed"
    except:
        print "Error"
        db.rollback()
    db.close()
    time.sleep(1)

Table Result:

+-----+------------+----------+------+
| id  | date       | time     | temp |
+-----+------------+----------+------+
| 107 | 2015-11-06 | 19:16:41 |   28 |
| 108 | 2015-11-06 | 19:16:42 |   28 |
| 109 | 2015-11-06 | 19:16:45 |   28 |
| 110 | 2015-11-06 | 19:16:46 |   28 |
| 111 | 2015-11-06 | 19:16:47 |   28 |
| 112 | 2015-11-06 | 19:16:48 |   28 |
| 113 | 2015-11-06 | 19:16:56 |   28 |
| 114 | 2015-11-06 | 19:17:00 |   28 |
| 115 | 2015-11-06 | 19:17:03 |   28 |
| 116 | 2015-11-06 | 19:17:05 |   28 |
| 117 | 2015-11-06 | 19:17:06 |   28 |
| 118 | 2015-11-06 | 19:17:07 |   28 |
| 119 | 2015-11-06 | 19:17:08 |   28 |
| 120 | 2015-11-06 | 19:17:09 |   28 |
| 121 | 2015-11-06 | 19:17:10 |   28 |
| 122 | 2015-11-06 | 19:17:11 |   28 |
+-----+------------+----------+------+

As you can see, sometimes the scrip insert data periodicaly, and sometimes we have 8 seconds of interval between the data. So, my question is: is possible to the interval between the data be 1 second every time? What am i doing wrong? Sorry for the bad english and thanks in advance!

Answer 1

You're establishing a new connection to the database server on each iteration. This can take arbitrary amount of time. Moving .connect() , etc. outside of the loop may give you more consistent timings:

db = MySQLdb.connect("localhost", "mauro", "12345", "temps")
curs = db.cursor()

while True:    
    try:
        curs.execute ("""INSERT INTO thetemps 
                values(0, CURRENT_DATE(), NOW(), 28)""")
        db.commit()
        print "Data committed"
    except:
        print "Error"
        db.rollback()    
    time.sleep(1)
db.close()

Answer 2

在尝试插入新行时，不要使用事务，也许某些表被锁定。

Answer 3

is possible to the interval between the data be 1 second every time?

Theoretically, yes, but in practice there're too many other factors outside of your control that are more likely to get in the way. Some of these include, but are not limited to:

1. OS kernel's task scheduler
2. Task priorities relative to others
3. Overall system load
4. Amount of data already in the table (check time complexity of binary trees)

This means that even if your system was idle most of the time, the time.sleep(1) is not guaranteed to always sleep for exactly 1 second, and even if it did, the system may've been doing something else (eg more I/O) and require different amounts of time to perform the same operations every time.

Also, instead of creating a new connection every time inside the loop, you should keep the connection open and save the overhead.

What am i doing wrong?

I don't think you're doing anything particularly wrong here. The code looks OK, except for the extra overhead of creating a new connection every time --which you shouldn't . That aside, the issue here boils down to factors outside of your control.

That being said, there're some things you can do to improve your chances.

---

A few additional suggestions to improve performance

Storage Engine

In addition to avoiding the overhead of opening/closing the database connection on every iteration, you should check the storage engine used for the table. For example, depending on your MySQL version, the default might still be MyISAM , which requires table locking for writing.

In contrast, InnoDB only requires row locking when writing to the table, which should improve things if something else is using the table. If you find you're not using InnoDB , issue an alter table ... query to change the storage engine.

Auto-Commit instead of Transaction

Transactions are meant to group a set of 2 or more queries as a single unit, but you're submitting individual queries. Instead, you should configure MySQL to have automatic commits enabled, so that it doesn't have to wait for an explicit commit request after your query is submitted and executed, saving some communication overhead between the server and your client.

Influence the OS Scheduler by Increasing Your Priority

You can set a higher priority for your program in order for the scheduler to be more helpful here. It might also help doing the same thing for the database service/process.

Other user -level tasks could also have their priorities lowered a bit, if necessary.

Answer 4

尝试在while条件之前创建与db的连接以保持打开连接。

Answer 5

The problem I see is that connecting + inserting takes time, this will add up and your process will eventually behind.

What I would do is separate the data gathering (make sure you read your temperatures every second) from the data loading (data loading can take longer than a second if needed, but you won't fall behind).

So, if I were you, I'd have two separate scripts running in parallel and communicating through some simple, fast and reliable mechanism. A list in Redis would probably work great. You could also use something like ZMQ .

Something like this:

# gather.py
while True:
    temp = get_temp()
    redis.lpush('temps', pickle.dumps([temp, datetime.now]))
    time.sleep(1)

and..

# load.py
while True:
    # Block until new temps are available in redis
    data, key = redis.brpop('temps')
    # Get all temps queued
    datapoints = [data] +  redis.rpop('temps'):
    db = MySQLdb.connect("localhost", "mauro", "12345", "temps")
    curs=db.cursor()
    try:
        for p in datapoints:
            point = pickle.loads(p)
            curs.execute ("INSERT INTO thetemps (temp, timestamp) values({}, {})".format(point[0], point[1])
        db.commit()
        print "Data committed"
    except:
        print "Error"
        db.rollback()

You could add some improvements on the above like reusing the db connection, making sure you don't lose the temps if there's a DB error, using a namedtuple instead of an array for the datapoints, etc.