Dataproc：未找到连接到外部 PostgreSQL 的 Sqoop 作业的 SSL 证书

Question

I need to connect to PostgreSQL db via SSL. I received 2 certificates and 1 key -> sslrootcert=root.crt sslcert=postgresql.crt and sslkey=postgresql.key.der

Here is my import config from Dataproc:

import
-Dmapreduce.job.user.classpath.first=true
-Dhadoop.security.credential.provider.path=jceks://hdfs/secrets/prod/db.jceks
--connect=jdbc:postgresql://xxx.x.x/DBNAME?sslrootcert=root.crt&sslcert=postgresql.crt&sslkey=postgresql.key.der
--username=user1
--table=db1
--target-dir=gs://bucket
--delete-target-dir
--as-avrodatafile
--password-alias=password

The Dataproc coundn't find my SSL certificates: Error: java.lang.RuntimeException: java.lang.RuntimeException: org.postgresql.util.PSQLException: Could not open SSL certificate file postgresql.crt

How I could add certificates and key to dataproc to be sure that these certs be accessible in connection config.

Where I should store SSL certificates to use them in import config during run job :

--connect=jdbc:postgresql://xxx.x.x/DBNAME?sslrootcert=/???/??/root.crt&sslcert=/???/??/postgresql.crt&sslkey=/???/??/postgresql.key.der

Answer 1

I think you can add the server certificate or its root CA certificate to Dataproc VMs at ${JAVA_HOME}/jre/lib/security/cacerts , for Dataproc it is /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/lib/security/cacerts . You can use an init action or create a custom image to get the cert in the dir.

See this doc and this question for more details.