将 1100 万行从 Postgresql 导入到 Pandas/Python

Question

I am trying to load 11 million records from a PostgreSQL DB which is hosted on an AWS server. I have tried to use pandas read_sql, and I am getting the result in 4 hours. I have 32 GB of RAM on my laptop and also Core i7, 7th gen. I have also set the chunk size to 10000, but it does not improve the crazy time. I have looked at many articles online, and tried all of them but none of them speeds up my process. I want to ideally load this data under 20 minutes if possible or whatever is the shortest amount of time possible. I need this data in a dataframe so that I can do some merges with other files that I have, and if I can fetch the data in Python, I can automate my process. My code is displayed below:

from io import StringIO
import psycopg2
import psycopg2.sql as sql
import pandas as pd
import numpy as np
import time


connection = psycopg2.connect(user="abc",
                                      password="efg",
                                      host="123.amazonaws.com",
                                      port="5432",
                                      database="db")

date='2020-03-01'
columns= '"LastName","FirstName","DateOfBirth","PatientGender","Key"'

postgreSQL_select_Query = 'select ' +  columns + ' from "Table" where "CreatedDate"::date>=' + "'" + date + "'" + 'limit 11000000'


x=pd.read_sql_query(postgreSQL_select_Query, connection, index_col=None, coerce_float=True, params=None, parse_dates=None, chunksize=10000)

Please suggest what I can do to improve this code, and reduce runtime.

I am also attaching another code segment, that I am using to do this, but the same result as it is fetching the rows in HOURS. Any guidance would be greatly appreciated.

Second Approach:

# -*- coding: utf-8 -*-

@author: ssullah
"""
from io import StringIO
import psycopg2
import psycopg2.sql as sql
import pandas as pd
import numpy as np
import time

start = time.time()
print("Started")

#Retreiving records from DB
def getdata():  
    try:
        start = time.time()
        print("Started")
        connection = psycopg2.connect(user="a"
                                      password="as",
                                      host="aws",
                                      port="5432",
                                      database="as")


        cur= connection.cursor()

        date='2020-03-01'
        columns= '"LastName","FirstName","DateOfBirth","PatientGender","Key"'

        postgreSQL_select_Query = 'select ' +  columns + ' from "ALLADTS" where "CreatedDate"::date>=' + "'" + date + "'" + 'limit 11000000'

        cur = connection.cursor('cursor-name') # server side cursor
        cur.itersize = 10000 # how much records to buffer on a client
        cur.execute(postgreSQL_select_Query)

        mobile_records = cur.fetchall() 


    #Column names as per schema, defined above
        col_names=["LastName","FirstName","DateOfBirth","PatientGender","Key"]

    # Create the dataframe, passing in the list of col_names extracted from the description
        records = pd.DataFrame(mobile_records,col_names)

        return records;


    except (Exception, psycopg2.Error) as error :
        print ("Error while fetching data from PostgreSQL", error)

    finally:
        #closing database connection.
        if(connection):
            cursor.close()
            connection.close()
            print("PostgreSQL connection is closed")


records=getdata()
end = time.time()
print("The total time:", (end - start)/60, 'minutes')

Answer 1

Update:

Instead of loading data in Python, I decided to create a temporary table in postgresql using Python and load the new file from pandas to Postgresql. Once the table was populated using a query in python, I was able to query and get the desired output aas the final result back in the panda dataframe.

All of this took 1.4 minutes, and the same query takes 30 minutes to run in Pgadmin, so by leveraging Python, and doing the calculation using sql query written in Python, I was able to exponentially speed up the process, and at the same time not have to deal with 11 million records in my memory. Thank you for your advice.