使用python將數據復制到Vertica

Question

我使用python和vertica-python庫將數據COPY到Vertica DB

connection = vertica_python.connect(**conn_info)
vsql_cur = connection.cursor()

with open("/tmp/vertica-test-insert", "rb") as fs:
    vsql_cur.copy( "COPY table FROM STDIN DELIMITER ',' ", fs, buffer_size=65536)
    connection.commit()

盡管文件包含更多行，但它僅插入5行數據。 這可能與數據庫設置有關，還是某些客戶端問題？

Answer 1

很可能您的行被拒絕了。 假設您使用的是7.x，則可以添加：

[ REJECTED DATA {'path' [ ON nodename ] [, ...] | AS TABLE 'reject_table'} ]

您還可以在執行復制后查詢此內容以查看結果摘要：

SELECT GET_NUM_ACCEPTED_ROWS() , GET_NUM_REJECTED_ROWS() ;

Answer 2

該代碼對我有用：

對於JSON

# for json file
with open("D:/SampleCSVFile_2kb/tweets.json", "rb") as fs:
    my_file = fs.read().decode('utf-8')
    cur.copy( "COPY STG.unstruc_data FROM STDIN parser fjsonparser()", my_file)
    connection.commit()

對於CSV

# for csv file
with open("D:/SampleCSVFile_2kb/SampleCSVFile_2kb.csv", "rb") as fs:
    my_file = fs.read().decode('utf-8','ignore')
    cur.copy( "COPY STG.unstruc_data FROM STDIN PARSER FDELIMITEDPARSER (delimiter=',', header='false') ", my_file) # buffer_size=65536
    connection.commit()