無法從 Data Fusion 連接 Cloud SQL PostgreSQL

Question

嘗試從 Cloud Data Fusion 連接 Cloud SQL PotgreSQL 數據庫（兩個私有實例在同一個 VPC 中，未共享），如此處逐步描述的：

https://cloud.google.com/data-fusion/docs/how-to/reading-from-postgresql https://cloud.google.com/data-fusion/docs/how-to/create-private-ip

用這個postgres-socket-factory-1.3.0-jar-with-driver-and-dependencies.jar作為驅動，下載地址： https : //github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory/releases /標簽/v1.3.0

測試連接，牧馬人像這樣被破壞：

牧馬人壞了

但 Data Fusion Wranger 日志也​​描述了此錯誤：

2021-06-29 21:28:02,226 - WARN  [service-http-executor-1147:i.c.w.s.c.AbstractWranglerHandler@210] - Error processing POST /v3/namespaces/system/apps/dataprep/services/service/methods/contexts/default/connections/databases, resulting in a 500 response.
java.lang.ExceptionInInitializerError: null
    at com.google.api.services.sqladmin.SQLAdmin$Builder.build(SQLAdmin.java:7339) ~[na:na]
    at com.google.cloud.sql.core.CoreSocketFactory.createAdminApiClient(CoreSocketFactory.java:365) ~[na:na]
    at com.google.cloud.sql.core.CoreSocketFactory.getInstance(CoreSocketFactory.java:141) ~[na:na]
    at com.google.cloud.sql.core.CoreSocketFactory.connect(CoreSocketFactory.java:252) ~[na:na]
    at com.google.cloud.sql.postgres.SocketFactory.createSocket(SocketFactory.java:76) ~[na:na]
    at org.postgresql.core.PGStream.createSocket(PGStream.java:223) ~[na:na]
    at org.postgresql.core.PGStream.<init>(PGStream.java:95) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213) ~[na:na]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[na:na]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223) ~[na:na]
    at org.postgresql.Driver.makeConnection(Driver.java:465) ~[na:na]
    at org.postgresql.Driver.connect(Driver.java:264) ~[na:na]
    at io.cdap.wrangler.service.database.JDBCDriverShim.connect(JDBCDriverShim.java:44) ~[expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_292]
    at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[na:1.8.0_292]
    at io.cdap.wrangler.service.database.DatabaseHandler.loadAndExecute(DatabaseHandler.java:558) [expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.database.DatabaseHandler.lambda$listDatabases$4(DatabaseHandler.java:322) [expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.common.AbstractWranglerHandler.respond(AbstractWranglerHandler.java:200) ~[expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.database.DatabaseHandler.listDatabases(DatabaseHandler.java:316) [expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.database.DatabaseHandlercdb75ebbd71058e7f79bb5e06b7f47ea.lambda$listDatabases_Lio_cdap_cdap_api_service_http_HttpServiceRequest_Lio_cdap_cdap_api_service_http_HttpServiceResponder_Ljava_lang_String__V(Unknown Source) [na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:560) ~[na:na]
    at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.execute(ServiceHttpServer.java:230) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.service.http.AbstractHttpHandlerDelegator.execute(AbstractHttpHandlerDelegator.java:66) ~[na:na]
    at io.cdap.wrangler.service.database.DatabaseHandlercdb75ebbd71058e7f79bb5e06b7f47ea.listDatabases(Unknown Source) [na:na]
    at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
    at io.cdap.http.internal.HttpMethodInfo.invoke(HttpMethodInfo.java:87) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.cdap.http.internal.HttpDispatcher.channelRead(HttpDispatcher.java:45) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.cdap.http.internal.NonStickyEventExecutorGroup$NonStickyOrderedEventExecutor.run(NonStickyEventExecutorGroup.java:254) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.util.concurrent.UnorderedThreadPoolEventExecutor$NonNotifyRunnable.run(UnorderedThreadPoolEventExecutor.java:277) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_292]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
Caused by: java.lang.IllegalStateException: You are currently running with version 1.30.9 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the Cloud SQL Admin API library.
    at com.google.common.base.Preconditions.checkState(Preconditions.java:537) ~[guava-20.0.jar:na]
    at com.google.api.client.util.Preconditions.checkState(Preconditions.java:113) ~[google-http-client-1.34.2.jar:1.34.2]
    at com.google.api.services.sqladmin.SQLAdmin.<clinit>(SQLAdmin.java:44) ~[na:na]
    ... 43 common frames omitted

請注意由以下原因Caused by: java.lang.IllegalStateException: You are currently running with version 1.30.9 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the Cloud SQL Admin API library. Caused by: java.lang.IllegalStateException: You are currently running with version 1.30.9 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the Cloud SQL Admin API library.

更新：還嘗試使用來自相同結果的來源生成的 fat-jar。

使用來自管道 Studio 的相同驅動程序測試連接我收到此錯誤：

2021-06-30 08:39:25,091 - ERROR [studio-http-executor-49:i.c.p.d.b.s.AbstractDBSource@140] - Exception while performing getSchema
org.postgresql.util.PSQLException: The connection attempt failed.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:315) ~[na:na]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[na:na]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223) ~[na:na]
    at org.postgresql.Driver.makeConnection(Driver.java:465) ~[na:na]
    at org.postgresql.Driver.connect(Driver.java:264) ~[na:na]
    at io.cdap.plugin.db.JDBCDriverShim.connect(JDBCDriverShim.java:60) ~[database-commons-1.5.3.jar:na]
    at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_292]
    at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[na:1.8.0_292]
    at io.cdap.plugin.db.batch.source.AbstractDBSource.getConnection(AbstractDBSource.java:219) [database-commons-1.5.3.jar:na]
    at io.cdap.plugin.db.batch.source.AbstractDBSource.getSchema(AbstractDBSource.java:132) [database-commons-1.5.3.jar:na]
    at io.cdap.plugin.db.batch.source.AbstractDBSource.configurePipeline(AbstractDBSource.java:113) [database-commons-1.5.3.jar:na]
    at io.cdap.plugin.cloudsql.postgres.CloudSQLPostgreSQLSource.configurePipeline(CloudSQLPostgreSQLSource.java:57) [1625042353080-0/:na]
    at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:306) [cdap-etl-core-6.4.1.jar:na]
    at io.cdap.cdap.datapipeline.service.ValidationHandler.validateStage(ValidationHandler.java:161) [expanded.16249866968976.4.1.73eaa383-b419-44a0-be67-36b2647bbaa0.jar/:na]
    at io.cdap.cdap.datapipeline.service.ValidationHandler4b0e8282e7c248900dbfd7c39c0c3cc8.lambda$validateStage_Lio_cdap_cdap_api_service_http_HttpServiceRequest_Lio_cdap_cdap_api_service_http_HttpServiceResponder_Ljava_lang_String__V(Unknown Source) [na:na]
    at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.lambda$execute$0(ServiceHttpServer.java:228) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.lambda$execute$3(AbstractContext.java:515) ~[na:na]
    at io.cdap.cdap.data2.transaction.Transactions$CacheBasedTransactional.finishExecute(Transactions.java:224) ~[na:na]
    at io.cdap.cdap.data2.transaction.Transactions$CacheBasedTransactional.execute(Transactions.java:211) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:512) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:500) ~[na:na]
    at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.execute(ServiceHttpServer.java:228) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.service.http.AbstractHttpHandlerDelegator.execute(AbstractHttpHandlerDelegator.java:66) ~[na:na]
    at io.cdap.cdap.datapipeline.service.ValidationHandler4b0e8282e7c248900dbfd7c39c0c3cc8.validateStage(Unknown Source) [na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
    at io.cdap.http.internal.HttpMethodInfo.invoke(HttpMethodInfo.java:87) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.cdap.http.internal.HttpDispatcher.channelRead(HttpDispatcher.java:45) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.cdap.http.internal.NonStickyEventExecutorGroup$NonStickyOrderedEventExecutor.run(NonStickyEventExecutorGroup.java:254) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.util.concurrent.UnorderedThreadPoolEventExecutor$NonNotifyRunnable.run(UnorderedThreadPoolEventExecutor.java:277) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_292]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_292]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_292]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_292]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_292]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_292]
    at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_292]
    at org.postgresql.core.PGStream.createSocket(PGStream.java:231) ~[na:na]
    at org.postgresql.core.PGStream.<init>(PGStream.java:95) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213) ~[na:na]
    ... 42 common frames omitted

Answer 1

正如@Anthanh Pham Trinh 所確認的那樣，使用 postgres-socket-factory 1.2 及以下版本將修復 Google API 庫版本問題。

問題中提到的wrangler 測試連接問題是一個已知問題，可以在https://cdap.atlassian.net/browse/CDAP-17973 中跟蹤對此的修復。

---

編輯：20210729

要使用不同的驅動程序，而不是 Data Fusion 讓您下載的默認 v1.0.16。 你可以在這個github鏈接（包括1.2版）下載不同版本的cloud-sql-jdbc驅動。

下載 1.2 版驅動程序后：

• 開放雲數據融合實例

• 點擊“集線器”

• 點擊“驅動程序”

• 搜索“CloudSQL PostgreSQL JDBC Driver”選擇它

  

• 點擊“部署”

  

• 在cloud-sql-jdbc 驅動程序 github 鏈接中上傳您下載的驅動程序。 在這個例子中，我下載了postgres-socket-factory-1.2.0-jar-with-driver-and-dependencies.jar

  

• 單擊“下一步”，驅動程序配置將自動填充，因為它將使用上傳的驅動程序的詳細信息。

  

• 點擊“完成”

• 上傳完成后，將顯示上傳的驅動程序版本。

  

您也可以通過轉到“牧馬人”來檢查這一點

• 在“牧馬人”中，單擊“+ 添加連接”

• 點擊“數據庫”

  

• 這將顯示所有可用的數據庫連接，並且 PostgreSQL 1.2 版應該可用。