Saving a Pandas Dataframe to a PostgreSQL Server in AWS Redshift

Question

I am trying to save a pandas dataframe to a postgresql redshift server. What I did was I connected to the server, transformed the table into a dataframe, did some analysis, then I wanted to save it as a postgresql table back into the AWS redshift server.

and here is my code:

import pandas.io.sql as psql
import pandas as pd
import psycopg2
from sqlalchemy import create_engine
import datetime as dt
import warnings
import numpy as np
warnings.filterwarnings("ignore")
#importing dependecies

print("Importing Dependencies")

#setting up the connection
conn_string = "dbname='database_name' user='username' password='password'"
conn = psycopg2.connect(conn_string)
print("Connecting to Database")

#read each table as a data frame
df_1 = psql.read_sql("SELECT name, salary, date FROM A", conn)
df_2 = psql.read_sql("SELECT name, gender FROM B", conn)
df_3 = psql.read_sql("SELECT name, health, recovery_time FROM C", conn)

I did some data analysis here then tried to save my result via

#Step 16: Saving result to postgresql table
print('Saving Partial Result as PostgreSQL Table')
engine = create_engine('postgresql://username:password@xx.xxx.xx.xxx/database_name')
con = engine.connect()
df.to_sql('health_informatics', con, if_exist='replace')
print('Successfully Saved to PostgreSQL')

however I received an error stating that:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: no pg_hba.conf entry for host "xx.xxx.xx.xxx", user "username",database "database_name", SSL off

I have read that this is a configuration issue:

https://serverfault.com/questions/704810/postgresql-error-with-pg-hba-conf

https://github.com/PostgresApp/PostgresApp/issues/234

http://docs.sqlalchemy.org/en/latest/errors.html#error-e3q8

Unfortunately, I do not have access to the private server root. I am wondering if there are work around on this, I cannot save the data as a .csv file because these are millions and millions of row, and are sensitive health records related data.

Answer 1

pg_hba.conf

(psycopg2.OperationalError) FATAL:

You can see from exception error log that you are trying access you database without configuration before.

Firstly you need precreate your configuration file. The general format of the pg_hba.conf file is a set of records.

initdb

The default way to do it. Command initdb initialize your pg_hba.conf.

It is possible to place the authentication configuration file elsewhere. You can edit this option in the same file.