[英]Can't connect Cloud Data Fusion with Google Cloud SQL for PostgreSQL
My goal is to read data from Cloud SQL Postgres to BigQuery via a Cloud Data Fusion pipeline.我的目标是通过 Cloud Data Fusion 管道将数据从 Cloud SQL Postgres 读取到 BigQuery。
For this, I set up a Cloud Data Fusion instance and assigned the following two permissions to the service account: (see https://cloud.google.com/data-fusion/docs/how-to/create-instance#setting_up_permissions )为此,我设置了一个 Cloud Data Fusion 实例并为服务帐户分配了以下两个权限:(请参阅https://cloud.google.com/data-fusion/docs/how-to/create-instance#setting_up_permissions )
As a next step I connected myself to the Cloud Data Fusion Instance, and navigated to Wrangler -> Add Connection -> Database -> Google Cloud SQL for PostgreSQL
.下一步,我将自己连接到 Cloud Data Fusion Instance,并导航到
Wrangler -> Add Connection -> Database -> Google Cloud SQL for PostgreSQL
。
As the driver I uploaded the postgres-socket-factory-1.0.13-jar-with-dependencies.jar
which I downloaded here: https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory作为驱动程序,我上传了我在这里下载的
postgres-socket-factory-1.0.13-jar-with-dependencies.jar
: https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory
For the driver configuration, I set:对于驱动程序配置,我设置:
For the database connection, I set:对于数据库连接,我设置:
<PROJECT_NAME>:<REGION>:<INSTANCE_CONNECTION_NAME>
<PROJECT_NAME>:<REGION>:<INSTANCE_CONNECTION_NAME>
jdbc:postgresql://google/<DATABASE_NAME>?cloudSqlInstance=<INSTANCE_CONNECTION_NAME>&socketFactory=com.google.cloud.sql.postgres.SocketFactory
jdbc:postgresql://google/<DATABASE_NAME>?cloudSqlInstance=<INSTANCE_CONNECTION_NAME>&socketFactory=com.google.cloud.sql.postgres.SocketFactory
After clicking on Test Connection, I receive the org.postgresql.Driver error message.单击测试连接后,我收到org.postgresql.Driver错误消息。
Most likely you need a driver and the JDBC Socket Factory to get it to work properly. 您很可能需要一个驱动程序和JDBC Socket Factory才能使其正常工作。 You should bundle the postgres connector with the socket factory into a uber-jar and see if that solves the problem.
您应该将postgres连接器与套接字工厂捆绑到一个超级jar中,看看是否能解决问题。
Here is a quick (untested) pom that should accomplish this: 这是一个应该完成此任务的快速(未经测试的)pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>postgres-with-cloud-sql-socket-factory</artifactId>
<packaging>pom</packaging>
<version>0.0.1</version>
<dependencies>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.5</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>postgres-socket-factory</artifactId>
<version>1.0.13</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
@Sebastian Auberger, @Sebastian Auberger,
We had the same issue, basically Data Fusion creates a Service Account which tries to create a IAP Tunnel to the Cloud SQL Instance. 我们遇到了同样的问题,基本上Data Fusion创建了一个服务帐户,该帐户尝试创建到云SQL实例的IAP隧道。 We could get through this issue by giving the role "Cloud SQL Client".
我们可以通过赋予角色“Cloud SQL Client”来解决这个问题。
Give it a shot and let us know!. 试一试,让我们知道!
On a related note, if the Data Fusion is private, then to reach the Cloud SQL instance through a private interface (no public IP traffic), you'll need a "Proxy VM", similar to what's described in this doc , but with a different startup-script, this blog post has a good overview on how to achieve this.在相关说明中,如果 Data Fusion 是私有的,那么要通过私有接口(没有公共 IP 流量)访问 Cloud SQL 实例,您需要一个“代理 VM”,类似于本文档中描述的内容,但具有一个不同的启动脚本, 这篇博文很好地概述了如何实现这一点。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.