讀取 .sql 文件以在 Python 中執行（pymysql）

Question

我正在嘗試創建一個多腳本工具，它將采用 .sql 文件的參數並執行它。

我設置了一個簡單的測試，只在一個數據庫上執行，但是語法每次都會給我帶來問題。

DELIMITER $$
CREATE DEFINER=`a_user`@`%` PROCEDURE `a_procedure`(
    IN DirectEmployeeID TEXT,
    IN StartRange DATE,
    IN EndRange DATE
)
BEGIN
SELECT aColumn
WHERE thisThing = 1;
END$$
DELIMITER ;

需要明確的是，這個腳本已經過測試，並且在通過時可以工作：

mysql -uuser -p -hhost -Pport databaseName < file.sql

也可以通過 mysql 工作台工作。

我在另一個網站上看到了這種類型的解決方案：

with conn.cursor() as cursor:
    f = sys.argv[1]
    file = open(f, 'r')
    sql = " ".join(file.readlines())
    cursor.execute(sql)

這給了我一個 MySQL 語法錯誤：

pymysql.err.ProgrammingError: (1064, u"You have an error in your SQL syntax; 
check the manual that corresponds to your MySQL server version for the right 
syntax to use near 'DELIMITER $$\n CREATE DEFINER=`a_user`@`%` PROCEDURE 
`MyCommissionsDirect`(\n \tIN ' at line 1")

如您所見，腳本中有 mysql 不喜歡的換行符。

然后我嘗試了這個：

with conn.cursor() as cursor:
    f = sys.argv[1]
    file = open(f, 'r')
    sql = ''
    line = file.readline()
    while line:
        sql += ' ' + line.strip('\n').strip('\t')
        line = file.readline()
    print sql
    cursor.execute(sql)

並得到另一個語法問題，打印顯示這都是一行，這在 mysqlworkbench 中不起作用。 甚至不嘗試執行它，這很奇怪。

當我首先將 DELIMETER $$ 放在單獨的行上時，它會在 mysqlworkbench 中執行。

這是我覺得我可能會讓事情變得越來越復雜的情況之一。 我很驚訝 pymysql 沒有直接執行 sql 文件的方法。 我厭倦了嘗試進行字符串操作並使其適用於這個特定的文件，因為這樣使這個工具模棱兩可和可重用的夢想就破滅了。

我是否以完全不正確的方式解決這個問題？

謝謝！

Answer 1

這是我使用PyMySQL的SQL文件的解決方案。 這些文件包含許多請求結束; 用於在列表中拆分請求。 所以要小心失蹤; 在列表中。

我決定補充遺失; 不在spar for for循環的函數中。 也許有更好的方法。

create-db-loff.sql ：

DROP DATABASE IF EXISTS loff;
CREATE DATABASE loff CHARACTER SET 'utf8';
USE loff;

CREATE TABLE product(
    `id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
    `code` BIGINT UNSIGNED NOT NULL UNIQUE,
    `name` VARCHAR(200),
    `nutrition_grades` VARCHAR(1)
);

CREATE TABLE category(
    `id`INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
    `name` VARCHAR(200)
);

CREATE TABLE asso_prod_cat(
    `category_id` INT UNSIGNED NOT NULL,
    `product_id` INT UNSIGNED NOT NULL,
    CONSTRAINT `fk_asso_prod_cat_category`
        FOREIGN KEY(category_id)
        REFERENCES category(id)
        ON DELETE CASCADE,
    CONSTRAINT `fk_asso_prod_cat_product`
        FOREIGN KEY(product_id)
        REFERENCES product(id)
        ON DELETE CASCADE
);

db.py ：

DB_CONFIG = {
    'host': 'localhost',
    'user': 'loff',
    'pass': 'loff',
    'db': 'loff',
    'char': 'utf8',
    'file': 'create-db-loff.sql'
}

def get_sql_from_file(filename=DB_CONFIG['file']):
    """
    Get the SQL instruction from a file

    :return: a list of each SQL query whithout the trailing ";"
    """
    from os import path

    # File did not exists
    if path.isfile(filename) is False:
        print("File load error : {}".format(filename))
        return False

    else:
        with open(filename, "r") as sql_file:
            # Split file in list
            ret = sql_file.read().split(';')
            # drop last empty entry
            ret.pop()
            return ret

request_list = self.get_sql_from_file()

if request_list is not False:

    for idx, sql_request in enumerate(request_list):
        self.message = self.MSG['request'].format(idx, sql_request)
        cursor.execute(sql_request + ';')

Answer 2

DELIMITER是MySQL解釋器使用的命令，例如命令行或Workbench，而不是實際的MySQL命令。

我最終在我的Python應用程序中使用某些邏輯來在定義DELIMITER時禁用MySQL查詢的執行，然后在再次定義DELIMITER時執行：

import MySQLdb
import re

file = 'somesql.sql'
conn = MySQLdb.Connection(mysqlserver, mysqluser, mysqlpass, mysqldb)
curs = conn.cursor()
ignorestatement = False # by default each time we get a ';' that's our cue to execute.
statement = ""
for line in open(file):
    if line.startswith('DELIMITER'):
        if not ignorestatement:
            ignorestatement = True # disable executing when we get a ';'
            continue
        else:
            ignorestatement = False # re-enable execution of sql queries on ';'
            line = " ;" # Rewrite the DELIMITER command to allow the block of sql to execute
    if re.match(r'--', line):  # ignore sql comment lines
        continue
    if not re.search(r'[^-;]+;', line) or ignorestatement:  # keep appending lines that don't end in ';' or DELIMITER has been called
        statement = statement + line
    else:  # when you get a line ending in ';' then exec statement and reset for next statement providing the DELIMITER hasn't been set
        statement = statement + line
        # print "\n\n[DEBUG] Executing SQL statement:\n%s" % (statement)
        try:
            curs.execute(statement)
            conn.commit()
            statement = ""
        except curs.Error, e:
            print(file + " - Error applying (" + str(e) + ")\nTerminating.")
            sys.exit(1)

這有點hacky，但似乎運作良好。

Answer 3

這是簡單的代碼

import pymysql

class ScriptRunner:

def __init__(self, connection, delimiter=";", autocommit=True):
    self.connection = connection
    self.delimiter = delimiter
    self.autocommit = autocommit

def run_script(self, sql):
    try:
        script = ""
        for line in sql.splitlines():
            strip_line = line.strip()
            if "DELIMITER $$" in strip_line:
                self.delimiter = "$$"
                continue
            if "DELIMITER ;" in strip_line:
                self.delimiter = ";"
                continue

            if strip_line and not strip_line.startswith("//") and not strip_line.startswith("--"):
                script += line + "\n"
                if strip_line.endswith(self.delimiter):
                    if self.delimiter == "$$":
                        script = script[:-1].rstrip("$") + ";"
                    cursor = self.connection.cursor()
                    print(script)
                    cursor.execute(script)
                    script = ""

        if script.strip():
            raise Exception("Line missing end-of-line terminator (" + self.delimiter + ") => " + script)

        if not self.connection.get_autocommit():
            self.connection.commit()
    except Exception:
        if not self.connection.get_autocommit():
            self.connection.rollback()
        raise

if __name__ == '__main__':
    connection = pymysql.connect(host="127.0.0.1", user="root", password="root", db="test", autocommit=True)
    sql = ""
    ScriptRunner(connection).run_script(sql)

Answer 4

大多數 SQL 文件包含解釋器命令，例如 DELIMITER，這使得將命令傳遞給 pymysql 有點困難，這個代碼片段允許您將 sql 文件中的語句分離到一個列表中以便順序執行。

def parse_sql(filename):
    data = open(filename, 'r').readlines()
    stmts = []
    DELIMITER = ';'
    stmt = ''

    for lineno, line in enumerate(data):
        if not line.strip():
            continue

        if line.startswith('--'):
            continue

        if 'DELIMITER' in line:
            DELIMITER = line.split()[1]
            continue

        if (DELIMITER not in line):
            stmt += line.replace(DELIMITER, ';')
            continue

        if stmt:
            stmt += line
            stmts.append(stmt.strip())
            stmt = ''
        else:
            stmts.append(line.strip())
    return stmts