如果使用Python在Postgres中主键或ID相同，如何附加值

Question

I'm trying to insert around 50 millions of data into postgresql using python script. I've file which contains 50 millions records. I'm completely new to PostgreSQL and Python as well. I tried below code to insert in python and I'm facing one challenge here. My test.txt contains a key-value pair like below.

If same key appears twice in the text file, I want to append the value with existing one. Which I'm not sure how to do that in python. Can you please some one help?

myfile.txt

key1 item1,product1,model1,price1|
key2 item2,product2,model2,price2|
key3 item3,product3,model3,price3|
key4 item4,product4,model4,price4|
key2 item22,product22,model22,price22|

In this case key2 has two records - while inserting into DB I've to append the second value with first one.

Tabular column:

key  value
key1 item1,product1,model1,price1|
key2 item2,product2,model2,price2|item22,product22,model22,price22|
key3 item3,product3,model3,price3|
key4 item4,product4,model4,price4|

insert.py

import psycopg2

def insertToDB(fileName):
  conn = psycopg2.connect("dbname='mydb' user='testuser' host='localhost'")
  with open(fileName) as f:
     for line in f:
       k,v = line.split(' ',1)
       cursor = conn.cursor()
       query = "INSERT INTO mytable (key,value) VALUES (%s,%s);"
       data = (key,value)
       cursor.execute(query,data)
       conn.commit()

 insertfile('myfile.txt')

I've around 50 millions of data and most of the key might have repeated key with different record, how to handle that and how efficiently we can write into DB?

It would be really helpful if someone can suggest to improvise this?

Thank you!

Answer 1

The easiest way is to use the ON CONFLICT clause of the SQL insert statement. This changes your simple insert into a "upsert" (insert or update).

ON CONFLICT requires PostgreSQL version 9.5 or greater, and is used like this:

query = """INSERT INTO mytable (key,value)
           VALUES (%s,%s)
           ON CONFLICT (key)
           DO UPDATE SET value = CONCAT(users.value, %s);"""
cursor.execute(query, (key, value, value))

The other option is to concatenate your results before you send them to the database by refactoring your data. Here I am collecting all rows by key in a dictionary, and then when inserting I'll just join all the values together.

This way, you only have one insert for each key.

Here is some code to explain this:

from collections import defaultdict
import psycopg2

def get_records(filename):
   records = defaultdict(list)
   with open(filename) as f:
     for line in f:
        if line.strip():
          key, value = line.split(' ',1)
          records[key].append(value)
   return records

def insert_records(records, conn):
   q = "INSERT INTO mytable (key, value) VALUES (%s, %s);"
   cursor = conn.cursor()
   for key, data in records.items():
      cursor.execute(q, (key, ''.join(data)))
      conn.commit()

conn = psycopg2.connect("dbname='mydb' user='testuser' host='localhost'")
insert_records(get_records('myfile.txt'), conn)

If you have a very large number of records, it may be that your are exhausting the memory by loading the entire file at once.

Instead, you can implement a simpler algorithm that keeps track of keys that are read.

def insert_records(filename, conn):
   seen = set()
   cursor = conn.cursor()
   qi = "INSERT INTO mytable (key, value) VALUES (%s, %s);"
   qu = "UPDATE mytable SET value = CONCAT(value, %s) WHERE key = %s;"

   with open(filename) as f:
     for line in f:
       if line.strip():
         key, value = line.split(' ', 1)
         if key not in seen:
            # first time we see this key, do an insert
            seen.add(key)
            cursor.execute(qi, (key, value))
         else:
            # key has been processed at least once, do an update
            cursor.execute(qu, (value, key))

         conn.commit()

conn = psycopg2.connect("dbname='mydb' user='testuser' host='localhost'")
insert_records(filename, conn)