[英]Apache Spark & JDBC : Socket Exception : Connection Reset
Our Spark Java application, task got an exception "com.microsoft.sqlserver.jdbc.SQLServerException: java.net.Socket Exception : Connection Reset"
, while it was running.我们的 Spark Java 应用程序任务在运行时出现异常"com.microsoft.sqlserver.jdbc.SQLServerException: java.net.Socket Exception : Connection Reset"
。
It makes a connection with database using following code, and table has millions of records:它使用以下代码与数据库建立连接,并且表有数百万条记录:
session.read().format("jdbc")
.option("url", dbProperties.getProperty("URL"))
.option("driverClass", dbProperties.getProperty("DRIVERCLASS"))
.option("username", dbProperties.getProperty("USERNAME"))
.option("password", dbProperties.getProperty("PASSWORD"))
.option("dbtable", "(" + formattedSQL + ") as " + tablenameDS)
.load();
Is there a way in Apache Spark, that it performs some X connection retries to the database with a delay between retries?在 Apache Spark 中是否有一种方法可以在重试之间延迟对数据库执行一些 X 连接重试?
Will increasing the following property " spark.task.maxFailures
" default value from 4, fix this issue?是否将以下属性“ spark.task.maxFailures
”默认值从 4 增加,以解决此问题?
Two solutions:两种解决方案:
Add JDBC-specific properties as options in option()
, like connectRetryCount
.在option()
添加特定于 JDBC 的属性作为选项,例如connectRetryCount
。 You can check the list at: https://docs.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver15 .您可以在以下位置查看列表: https : //docs.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver15 。
Add the JDBC properties in the URL itself.在 URL 本身中添加 JDBC 属性。
Your code could look like:您的代码可能如下所示:
session.read().format("jdbc")
.option("connectRetryCount", 200)
.option("url", dbProperties.getProperty("URL"))
.option("driverClass", dbProperties.getProperty("DRIVERCLASS"))
.option("username", dbProperties.getProperty("USERNAME"))
.option("password", dbProperties.getProperty("PASSWORD"))
.option("dbtable", "(" + formattedSQL + ") as " + tablenameDS)
.load();
You can find detailed examples there: https://github.com/jgperrin/net.jgp.books.spark.ch08/tree/master/src/main/java/net/jgp/books/spark/ch08/lab100_mysql_ingestion .你可以在那里找到详细的例子: https : //github.com/jgperrin/net.jgp.books.spark.ch08/tree/master/src/main/java/net/jgp/books/spark/ch08/lab100_mysql_ingestion 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.