使用 pyspark 连接到 Presto SQL 目录并在 postgresql 数据库上执行查询的 Pyspark 代码或步骤？

Question

I have pyspark configured to work with PostgreSQL directly. However, I want to pass data from spark to presto using jdbc connector, and then run the query on postgresql using pyspark and presto. How can I do that code-wise?

from pyspark.sql import SparkSession
from pyspark import SparkContext,SparkConf
from pyspark.sql import SQLContext
import sys
sys.path.append('/usr/local/lib/python3.6/dist-packages')
import requests
import json, ast

sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
spark = SparkSession.builder \
.master("local") \
.appName("jdbc data sources") \
.config("spark.sql.shuffle.partitions", "4") \
.getOrCreate()
driver = "io.prestosql.jdbc.PrestoDriver" 
#path = "//host:port/prestosql/?user=<username>&password=<passwd>"
path = "//host:port/prestosql<catalog>"
url = "jdbc:presto:" + path
tablename = <tablename>
dbDataFrame = spark.read.format("jdbc").option("url", url).option("dbtable", "<select query>").option("driver", driver).load()

What am I doing wrong? I want to run a select query on postgresql via presto and pass the result back to spark using pyspark.

I am getting following error :

in get_return_value py4j.protocol.Py4JJavaError: An error occurred while 
calling o53.load. : java.sql.SQLException: Authentication using 
username/password requires SSL to be enabled at 
io.prestosql.jdbc.PrestoDriverUri.setupClient(PrestoDriverUri.java:160) at 
io.prestosql.jdbc.PrestoDriver.connect(PrestoDriver.java:91) at  org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)

When i enable .option("SSL","true"), I am getting new error :

py4j.protocol.Py4JJavaError: An error occurred while calling o84.load. : 
java.sql.SQLException: Error executing query at 
io.prestosql.jdbc.PrestoStatement.internalExecute(PrestoStatement.jav a:284) 
at io.prestosql.jdbc.PrestoStatement.execute(PrestoStatement.java:229) at 
io.prestosql.jdbc.PrestoPreparedStatement.<init>(PrestoPreparedStatem 
ent.java:80

What am i doing wrong.. pls help

Answer 1

我猜您的 sql 查询中可能存在错误。首选语法类似于.option("dbtable","(select * from sample_table)a").load()