How to connect to Greenplum Database remotely from PySpark in Jupyter Notebook?

Question

I'm trying to connect PySpark (with Jupyter Notebook) to a Greenplum Database instance on Oracle VM VirtualBox through JDBC connection, however I'm receiving the following error when I KNOW the password is correct.:

Py4JJavaError: An error occurred while calling o424.load.
: org.postgresql.util.PSQLException: FATAL: password authentication failed 
for user "user2"

I've tried:

Reviewing Greenplum DB docs regarding connecting with PySpark

Changing Postgresql connectivity setting in gp_hba.conf, sshd_conf, and postgresql.conf files

Utilizing pyspark shell and loading .jar file as

pyspark --jars 'path to .jar file'

then running mentioned code below

PySpark code in Jupyter Notebook is:

import findspark
findspark.init('spark-2.4.1-bin-hadoop2.7')
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

option = {
    'url':"jdbc:postgresql://localhost:5432/tutorial",
    'user':"user2",
    'password':"SECRET",
    'dbschema':"faa",
    'dbtable':"otp_c",
    'partitionColumn':"airlineid"
}

gpdf = spark.read.format('greenplum').options(**option).load()

Pivotal Greenplum instructs having a connector .jar file for JDBC connection into the database, which I have located in spark-2.4.1-bin-hadoop2.7/jars/greenplum-spark_2.11-1.6.0.jar

Additionally, within the Greenplum DB, the gp_hba.conf is configured as:

# If you want to allow non-local connections, you need to add more
# "host" records.  In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.
# CAUTION: Configuring the system for local "trust" authentication allows
# any local user to connect as any PostgreSQL user, including the database
# superuser. If you do not trust all your local users, use another
# authentication method.
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
# IPv4 local connections:
# IPv6 local connections:
local    all         gpadmin         ident
host     all         gpadmin         127.0.0.1/28    trust
host     all         gpadmin         10.0.2.15/32       trust
host     all         gpadmin         ::1/128       trust
host     all         gpadmin   fe80::a00:27ff:fe84:1f3f/128      trust
local    replication gpadmin         ident
host     replication gpadmin         samenet       trust
local    gpperfmon         gpmon         md5
host     all         gpmon         127.0.0.1/28    md5
local    tutorial            +users     trust
host     tutorial            +users     trust
host all all 0.0.0.0/0 md5
#local all all md5
#local all user2 ident

The sshd_config file is configured with

PasswordAuthentication yes

Finally, the postgresql.conf file is configured with

# - Connection Settings -

listen_addresses = '*'          # what IP address(es) to listen on;
                                    # comma-separated list of addresses;
                                    # defaults to '*', '*' = all
                                    # (change requires restart)

port=5432 ##port = 5432                         # sets the database 
listener port for
                                    # a Greenplum instance. The master and
                                    # each segment has its own port 
number.
# note: Port numbers for the Greenplum system must also be changed in the
# gp_configuration catalog. See the Greenplum Database Administrator Guide
# for instructions!
#
#

I'm expecting to connect to Greenplum DB and perform SQL queries with PySpark however I receive Py4JJavaError.

Unsure what other options exist, ideally I want to connect via Jupyter Notebook please help!

Answer 1

In pg_hba, host config needs CIDR. The line

host tutorial +users trust

won't take effect. So it falls through the last line and ask for password.

You can create a role user2 with a secret password inside greenplum cluster.