堆栈内存溢出
  简体   繁体   English

无法从 Data Fusion 连接 Cloud SQL PostgreSQL

[英]Could not connect Cloud SQL PostgreSQL from Data Fusion

 原文  2021-06-29 21:43:54  7   1    cloud/ google-cloud-sql/ google-api-client/ google-cloud-data-fusion

问题描述

Trying to connect a Cloud SQL PotgreSQL database from a Cloud Data Fusion (both private instances in the same VPC, not shared) as described here step by step:尝试从 Cloud Data Fusion 连接 Cloud SQL PotgreSQL 数据库（两个私有实例在同一个 VPC 中，未共享），如此处逐步描述的：

https://cloud.google.com/data-fusion/docs/how-to/reading-from-postgresql https://cloud.google.com/data-fusion/docs/how-to/create-private-ip https://cloud.google.com/data-fusion/docs/how-to/reading-from-postgresql https://cloud.google.com/data-fusion/docs/how-to/create-private-ip

With this postgres-socket-factory-1.3.0-jar-with-driver-and-dependencies.jar as driver, downloaded from: https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory/releases/tag/v1.3.0用这个postgres-socket-factory-1.3.0-jar-with-driver-and-dependencies.jar作为驱动，下载地址： https : //github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory/releases /标签/v1.3.0

Testing the connection, wrangler gets broken like this:测试连接，牧马人像这样被破坏：

wrangler broken牧马人坏了

But also Data Fusion Wranger logs describes this error:但 Data Fusion Wranger 日志也​​描述了此错误：

2021-06-29 21:28:02,226 - WARN  [service-http-executor-1147:i.c.w.s.c.AbstractWranglerHandler@210] - Error processing POST /v3/namespaces/system/apps/dataprep/services/service/methods/contexts/default/connections/databases, resulting in a 500 response.
java.lang.ExceptionInInitializerError: null
    at com.google.api.services.sqladmin.SQLAdmin$Builder.build(SQLAdmin.java:7339) ~[na:na]
    at com.google.cloud.sql.core.CoreSocketFactory.createAdminApiClient(CoreSocketFactory.java:365) ~[na:na]
    at com.google.cloud.sql.core.CoreSocketFactory.getInstance(CoreSocketFactory.java:141) ~[na:na]
    at com.google.cloud.sql.core.CoreSocketFactory.connect(CoreSocketFactory.java:252) ~[na:na]
    at com.google.cloud.sql.postgres.SocketFactory.createSocket(SocketFactory.java:76) ~[na:na]
    at org.postgresql.core.PGStream.createSocket(PGStream.java:223) ~[na:na]
    at org.postgresql.core.PGStream.<init>(PGStream.java:95) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213) ~[na:na]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[na:na]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223) ~[na:na]
    at org.postgresql.Driver.makeConnection(Driver.java:465) ~[na:na]
    at org.postgresql.Driver.connect(Driver.java:264) ~[na:na]
    at io.cdap.wrangler.service.database.JDBCDriverShim.connect(JDBCDriverShim.java:44) ~[expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_292]
    at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[na:1.8.0_292]
    at io.cdap.wrangler.service.database.DatabaseHandler.loadAndExecute(DatabaseHandler.java:558) [expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.database.DatabaseHandler.lambda$listDatabases$4(DatabaseHandler.java:322) [expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.common.AbstractWranglerHandler.respond(AbstractWranglerHandler.java:200) ~[expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.database.DatabaseHandler.listDatabases(DatabaseHandler.java:316) [expanded.16249867015644.4.1.86000cc8-2950-4df3-8ded-ef6513642958.jar/:na]
    at io.cdap.wrangler.service.database.DatabaseHandlercdb75ebbd71058e7f79bb5e06b7f47ea.lambda$listDatabases_Lio_cdap_cdap_api_service_http_HttpServiceRequest_Lio_cdap_cdap_api_service_http_HttpServiceResponder_Ljava_lang_String__V(Unknown Source) [na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:560) ~[na:na]
    at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.execute(ServiceHttpServer.java:230) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.service.http.AbstractHttpHandlerDelegator.execute(AbstractHttpHandlerDelegator.java:66) ~[na:na]
    at io.cdap.wrangler.service.database.DatabaseHandlercdb75ebbd71058e7f79bb5e06b7f47ea.listDatabases(Unknown Source) [na:na]
    at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
    at io.cdap.http.internal.HttpMethodInfo.invoke(HttpMethodInfo.java:87) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.cdap.http.internal.HttpDispatcher.channelRead(HttpDispatcher.java:45) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.cdap.http.internal.NonStickyEventExecutorGroup$NonStickyOrderedEventExecutor.run(NonStickyEventExecutorGroup.java:254) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.util.concurrent.UnorderedThreadPoolEventExecutor$NonNotifyRunnable.run(UnorderedThreadPoolEventExecutor.java:277) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_292]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
Caused by: java.lang.IllegalStateException: You are currently running with version 1.30.9 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the Cloud SQL Admin API library.
    at com.google.common.base.Preconditions.checkState(Preconditions.java:537) ~[guava-20.0.jar:na]
    at com.google.api.client.util.Preconditions.checkState(Preconditions.java:113) ~[google-http-client-1.34.2.jar:1.34.2]
    at com.google.api.services.sqladmin.SQLAdmin.<clinit>(SQLAdmin.java:44) ~[na:na]
    ... 43 common frames omitted

Notice the Caused by: java.lang.IllegalStateException: You are currently running with version 1.30.9 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the Cloud SQL Admin API library.请注意由以下原因Caused by: java.lang.IllegalStateException: You are currently running with version 1.30.9 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the Cloud SQL Admin API library. Caused by: java.lang.IllegalStateException: You are currently running with version 1.30.9 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the Cloud SQL Admin API library.

Update: Also tried with a genererated fat-jar from sources with same result.更新：还尝试使用来自相同结果的来源生成的 fat-jar。

Testing conection with same driver from pipeline Studio I get this error:使用来自管道 Studio 的相同驱动程序测试连接我收到此错误：

2021-06-30 08:39:25,091 - ERROR [studio-http-executor-49:i.c.p.d.b.s.AbstractDBSource@140] - Exception while performing getSchema
org.postgresql.util.PSQLException: The connection attempt failed.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:315) ~[na:na]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[na:na]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223) ~[na:na]
    at org.postgresql.Driver.makeConnection(Driver.java:465) ~[na:na]
    at org.postgresql.Driver.connect(Driver.java:264) ~[na:na]
    at io.cdap.plugin.db.JDBCDriverShim.connect(JDBCDriverShim.java:60) ~[database-commons-1.5.3.jar:na]
    at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_292]
    at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[na:1.8.0_292]
    at io.cdap.plugin.db.batch.source.AbstractDBSource.getConnection(AbstractDBSource.java:219) [database-commons-1.5.3.jar:na]
    at io.cdap.plugin.db.batch.source.AbstractDBSource.getSchema(AbstractDBSource.java:132) [database-commons-1.5.3.jar:na]
    at io.cdap.plugin.db.batch.source.AbstractDBSource.configurePipeline(AbstractDBSource.java:113) [database-commons-1.5.3.jar:na]
    at io.cdap.plugin.cloudsql.postgres.CloudSQLPostgreSQLSource.configurePipeline(CloudSQLPostgreSQLSource.java:57) [1625042353080-0/:na]
    at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:306) [cdap-etl-core-6.4.1.jar:na]
    at io.cdap.cdap.datapipeline.service.ValidationHandler.validateStage(ValidationHandler.java:161) [expanded.16249866968976.4.1.73eaa383-b419-44a0-be67-36b2647bbaa0.jar/:na]
    at io.cdap.cdap.datapipeline.service.ValidationHandler4b0e8282e7c248900dbfd7c39c0c3cc8.lambda$validateStage_Lio_cdap_cdap_api_service_http_HttpServiceRequest_Lio_cdap_cdap_api_service_http_HttpServiceResponder_Ljava_lang_String__V(Unknown Source) [na:na]
    at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.lambda$execute$0(ServiceHttpServer.java:228) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.lambda$execute$3(AbstractContext.java:515) ~[na:na]
    at io.cdap.cdap.data2.transaction.Transactions$CacheBasedTransactional.finishExecute(Transactions.java:224) ~[na:na]
    at io.cdap.cdap.data2.transaction.Transactions$CacheBasedTransactional.execute(Transactions.java:211) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:512) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:500) ~[na:na]
    at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.execute(ServiceHttpServer.java:228) ~[na:na]
    at io.cdap.cdap.internal.app.runtime.service.http.AbstractHttpHandlerDelegator.execute(AbstractHttpHandlerDelegator.java:66) ~[na:na]
    at io.cdap.cdap.datapipeline.service.ValidationHandler4b0e8282e7c248900dbfd7c39c0c3cc8.validateStage(Unknown Source) [na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
    at io.cdap.http.internal.HttpMethodInfo.invoke(HttpMethodInfo.java:87) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.cdap.http.internal.HttpDispatcher.channelRead(HttpDispatcher.java:45) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at io.cdap.http.internal.NonStickyEventExecutorGroup$NonStickyOrderedEventExecutor.run(NonStickyEventExecutorGroup.java:254) ~[io.cdap.http.netty-http-1.5.0.jar:na]
    at io.netty.util.concurrent.UnorderedThreadPoolEventExecutor$NonNotifyRunnable.run(UnorderedThreadPoolEventExecutor.java:277) ~[io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_292]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_292]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_292]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_292]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_292]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_292]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_292]
    at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_292]
    at org.postgresql.core.PGStream.createSocket(PGStream.java:231) ~[na:na]
    at org.postgresql.core.PGStream.<init>(PGStream.java:95) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98) ~[na:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213) ~[na:na]
    ... 42 common frames omitted

1 个解决方案

解决方案1
 -1  2021-07-02 00:12:24

Using version 1.2 and below of postgres-socket-factory will fix Google API library version issue as confirmed by @Anthanh Pham Trinh.正如@Anthanh Pham Trinh 所确认的那样，使用 postgres-socket-factory 1.2 及以下版本将修复 Google API 库版本问题。

The wrangler test connection issue as mentioned in the question is a known issue and fix for this can be tracked in https://cdap.atlassian.net/browse/CDAP-17973 .问题中提到的wrangler 测试连接问题是一个已知问题，可以在https://cdap.atlassian.net/browse/CDAP-17973 中跟踪对此的修复。

EDIT: 20210729编辑：20210729

To use a different driver rather than the default v1.0.16 that Data Fusion let's you download.要使用不同的驱动程序，而不是 Data Fusion 让您下载的默认 v1.0.16。 You can download the different versions of cloud-sql-jdbc drivers in this github link(version 1.2 included) .你可以在这个github链接（包括1.2版）下载不同版本的cloud-sql-jdbc驱动。

When the version 1.2 driver is downloaded:下载 1.2 版驱动程序后：

  • Open Cloud Data Fusion Instance开放云数据融合实例

  • Click "HUB"点击“集线器”

  • Click "Drivers"点击“驱动程序”

  • Search for "CloudSQL PostgreSQL JDBC Driver" select it搜索“CloudSQL PostgreSQL JDBC Driver”选择它

    在此处输入图片说明

  • Click "Deploy"点击“部署”

    在此处输入图片说明

  • Upload the driver that you have downloaded in cloud-sql-jdbc drivers github link .在cloud-sql-jdbc 驱动程序 github 链接中上传您下载的驱动程序。 In this example I downloaded postgres-socket-factory-1.2.0-jar-with-driver-and-dependencies.jar在这个例子中，我下载了postgres-socket-factory-1.2.0-jar-with-driver-and-dependencies.jar

    在此处输入图片说明

  • Click "Next", driver configuration will be auto populated since it will use the details of the uploaded driver.单击“下一步”，驱动程序配置将自动填充，因为它将使用上传的驱动程序的详细信息。

    在此处输入图片说明

  • Click "Finish"点击“完成”

  • When upload is done, it will show the driver version that was uploaded.上传完成后，将显示上传的驱动程序版本。

    在此处输入图片说明

You can also check this by going to "Wrangler"您也可以通过转到“牧马人”来检查这一点

  • When in "Wrangler", click "+ Add Connection"在“牧马人”中，单击“+ 添加连接”

  • Click "Database"点击“数据库”

    在此处输入图片说明

  • This will show all the available database connections, and PostgreSQL version 1.2 should be available.这将显示所有可用的数据库连接，并且 PostgreSQL 1.2 版应该可用。

    在此处输入图片说明

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 无法从 Data Fusion 连接 Cloud SQL mySql/postgreSQL 实例 - Unable to connect Cloud SQL mySql / postgreSQL instance from Data Fusion 如何在 Cloud Data Fusion 上的 PostgreSQL 中执行 upsert 操作 - How to perform upsert operation in PostgreSQL on Cloud Data Fusion 如何在 Cloud Data Fusion 中将牧马人连接添加到云 sql postgres - How to add wrangler connection to cloud sql postgres in Cloud Data Fusion 使用 Cloud Data Fusion 从 MySql 在 Bigquery 中追加增量数据 - Appending incremental data in Bigquery from MySql using Cloud Data Fusion 从本地 Python IDE 连接云 Sql - cloud Sql connect from Local Python IDE 如何使用 PHP 从 Google Cloud SQL 连接和检索数据？ - How do I connect and retrieve data from Google Cloud SQL using PHP? 如何从其他服务器连接到Google Cloud SQL实例 - How to Connect to a Google Cloud SQL Instance from a Different Server 无法从Google App Engine节点连接到Cloud sql中的Postgres - Unable to connect to postgres in cloud sql from google app engine nodes 远程连接到Google Cloud SQL - Remote Connect to Google Cloud SQL 无法将云 function 连接到云 SQL - Cannot connect cloud function to Cloud SQL
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM