Spark MySQL java 連接器

Question

我想連接 spark 和 MySQL。 我嘗試了以下代碼：

public class Get_Data_From_MySQL implements Serializable {

private static final org.apache.log4j.Logger LOGGER = org.apache.log4j.Logger.getLogger(Get_Data_From_MySQL.class);

private static final String MYSQL_CONNECTION_URL = "jdbc:mysql://localhost:3306/test";
private static final String MYSQL_USERNAME = "root";
private static final String MYSQL_PWD = "";

private static final SparkSession sparkSession =
        SparkSession.builder().master("local[*]").appName("Spark2JdbcDs").getOrCreate();

public static void main(String[] args) {
    //JDBC connection properties
    final Properties connectionProperties = new Properties();
    connectionProperties.put("user", MYSQL_USERNAME);
    connectionProperties.put("password", MYSQL_PWD);
    connectionProperties.put("driver", "com.mysql.jdbc.Driver");
    final String dbTable =
            "(select age from employe";
      //Load MySQL query result as Dataset
    Dataset<Row> jdbcDF =
            sparkSession.read()
                    .jdbc(MYSQL_CONNECTION_URL, dbTable, "age", 10001, 499999, 10, connectionProperties);

我收到此錯誤：

Exception in thread "main" java.sql.SQLException: Communication link failure: Mauvais 'handshake'
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:659)

在這一行：

Dataset<Row> jdbcDF =sparkSession.read().jdbc(MYSQL_CONNECTION_URL, dbTable, "age", 10001, 499999, 10, connectionProperties);

我檢查了 MySQL 用戶和密碼，一切都正確。

謝謝你。

Answer 1

您面臨的問題可能意味着數據庫根本無法訪問。 This problem can occur due to any reason may be IP address or hostname in JDBC URL is wrong, or the hostname in JDBC URL is not recognized by local DNS server, or Port number may be missing, or DB server is down. 可能有很多原因，所以我建議您再次通過您的方法 go 並檢查所有內容。 通過終端連接並檢查

Answer 2

  package JavaSpark.Javs.SQL;

   import java.util.Properties;

   import org.apache.spark.sql.Dataset;
   import org.apache.spark.sql.Row;
   import org.apache.spark.sql.SparkSession;


 public class sparkSqlMysql {


private static final org.apache.log4j.Logger LOGGER = org.apache.log4j.Logger.getLogger(sparkSqlMysql.class);

private static final SparkSession sparkSession =
        SparkSession.builder().master("local[*]").appName("Spark2JdbcDs").getOrCreate();    

public static void main(String[] args) {
    //JDBC connection properties
    
    final Properties connectionProperties = new Properties();
    connectionProperties.put("user", "root");
    connectionProperties.put("password","mypassword");
    connectionProperties.put("driver", "com.mysql.jdbc.Driver");
     // Load MySQL query result as Dataset
    Dataset<Row> jdbcDF2 =
            sparkSession.read()
                    .jdbc("jdbc:mysql://localhost:3306/SQLprep", "customer", connectionProperties);
    
    jdbcDF2.show();
 }