Python将数组元素插入MySQL数据库

Question

I am processing images and converting into around 400 data values. I want each one of these values to be stored in a column. My mysql table has got columns like these:

MYID, WIDTH, HEIGHT,P1,P2,P3.....P400.

I can easily save these into a csv file, but since the processing happens on around 3 million files I thought I will write these output directly to a mysql table instead of creating multiple csv files.

This is what I have written so far:

for (i, imagePath) in enumerate(imagePaths):
    filename = imagePath[imagePath.rfind("/") + 1:]
    image = cv2.imread(imagePath)
    rows, cols, channels = image.shape
    if not image is None:
        features = detail.describe(image)
        features = [str(x) for x in features]
        fileparam = [filename,cols,rows]
        sqldata = fileparam+features
        var_string = ', '.join('?' * len(sqldata))
        query_string = 'INSERT INTO lastoneweeknew VALUES (%s)' % var_string
        y.execute(query_string, sqldata)

If I print sqldata, it gets printed like this:

['120546506.jpg',650, 420, '0.0', '0.010269055',........., '0.8539078']

The mysql table has these data types:

+----------+----------------+------+-----+---------+----------------+
| Field    | Type           | Null | Key | Default | Extra          |
+----------+----------------+------+-----+---------+----------------+
| image_id | int(11)        | NO   | PRI | NULL    | auto_increment |
| MYID     | int(10)        | YES  |     | NULL    |                |
| WIDTH    | decimal(6,2)   | YES  | MUL | NULL    |                |
| HEIGHT   | decimal(6,2)   | YES  | MUL | NULL    |                |
| P1       | decimal(22,20) | YES  |     | NULL    |                |
| P2       | decimal(22,20) | YES  |     | NULL    |                |

When I insert the data into mysql table I am getting the following error:

TypeError: not all arguments converted during string formatting

However, when I write the output to csv file and insert the csv data into mysql using R, I could insert without any trouble.

I thought the row and column values are integers and the rest looks like text in the output and hence I converted them to text.

row = str(rows)
col = str(cols)

But I am still getting the same error.

Answer 1

For your error - %s can only be used to format string arguments, but some of your arguments are int types - thus the type error.

It looks like you are trying to build a data frame and upload it to a MySQL database - luckily this is a common task so there is a library called pandas that can do this all for you. If you create a list of dictionaries where each of the dictionarys key-value pairs are ColumnName: Value.

import pandas as pd
from pandas.io import sql
import MySQLdb

def handlePaths(imagePaths):
    imageDataList = []
    for (i, imagePath) in enumerate(imagePaths):
        filename = imagePath[imagePath.rfind("/") + 1:]
        image = cv2.imread(imagePath)
        rows, cols, channels = image.shape
        if not image is None:
            features = detail.describe(image)
            features = [str(x) for x in features]
            fileparam = [filename,cols,rows]
            sqldata = fileparam+features
            imageData = {"MYID" : value,
             "WIDTH" : value,
             "HEIGHT": value,
             "P1": value, #I would do these iterivly 
             .....,
             "P400": value}
            imageDataList.append(imageData)
    imageDataFrame = pd.DataFrame(imageDataList)
    database_connection = MySQLdb.connect()  # may need to add some other options to connect
    imageDataFrame.to_sql(con=database_connection, name='lastoneweeknew', if_exists='replace')

I assume this is a pretty cpu-consuming process, you could assign an image to each cpu to make it run quicker, By uploaded each individual entry you let the database handle the race conditions.

import pandas as pd
from pandas.io import sql
import MySQLdb
import multiprocessing

def analyzeImages(imagePaths) #imagePaths is a list of image paths
    pool = multiprocessing.Pool(cpu_count)
    pool.map(handleSinglePath, imagePaths)
    pool.join()
    pool.close()

def handleSinglePath(imagePath):
    image = cv2.imread(imagePath) #Not sure what you where doing before here but you can do it again 
    rows, cols, channels = image.shape
    if not image is None:
        features = detail.describe(image)
        features = [str(x) for x in features]
        fileparam = [filename,cols,rows]
        sqldata = fileparam+features
        imageData = {"MYID" : value,
         "WIDTH" : value,
         "HEIGHT": value,
         "P1": value, #I would do these iterivly 
         .....,
         "P400": value}
    imageDataFrame = pd.DataFrame(imageData)
    database_connection = MySQLdb.connect()  # may need to add some other options to connect
    imageDataFrame.to_sql(con=database_connection, name='lastoneweeknew', if_exists='replace')