在 dataproc 集群上運行時無法從 spark jar 中創建 bigquery 連接

Question

我想從 dataproc 集群上運行的 Spark 中刪除 bigquery 表中的數據。 但是在運行 Spark 應用程序時出現 SIGSEGV 運行時錯誤。 這是嘗試時的完整錯誤：

 datasource.getConnection()

 A fatal error has been detected by the Java Runtime Environment:
 SIGSEGV (0xb) at pc=0x00007f218680fc80, pid=25451, tid=0x00007f21f8c20700

 JRE version: OpenJDK Runtime Environment (8.0_322-b06) (build 1.8.0_322-b06)
 Java VM: OpenJDK 64-Bit Server VM (25.322-b06 mixed mode linux-amd64 compressed oops)
 Problematic frame:
 C  0x00007f218680fc80
 Failed to write core dump. Core dumps have been disabled. To enable core dumping, try ulimit -c unlimited" before starting Java again
 An error report file with more information is saved as: /tmp/4f00321a-e1ae-4869-9540-ff789383b27c/hs_err_pid25451.log
 If you would like to submit a bug report, please visit: https://github.com/adoptium/adoptium-support/issues

這是我正在關注的示例代碼：

public class ConnectServiceAuthentication {
public static Connection connectViaDS() throws SQLException {
    Connection connection = null;
    DataSource ds = new 
com.simba.googlebigquery.jdbc.DataSource();
    ds.setURL("jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;\n" + "OAuthType=3;ProjectId=myProjectid;");
    ds.setProjectId("myProjectid");
    ds.setOAuthType(0); // Service Authentication
ds.setOAuthServiceAcctEmail("myproject_bigquery@myproject.iam.gserviceaccount.com");
    ds.setOAuthPvtKey("/home/key.json");
    connection = ds.getConnection();
    return connection;
}

public static void main(String[] args) throws SQLException {
    Connection connection = connectViaDS();
    String sql = "select count(1) from  temp.tempn1;";

    PreparedStatement ps = connection.prepareStatement(sql);
    ResultSet rs = ps.executeQuery();
    if (rs.next())
        System.out.println(rs.getInt(1));
    
    connection.close();
}

}

Answer 1

為什么不使用 java bigquery 客戶端？ 我正在使用我的 Spark 應用程序中的客戶端從 BigQuery 插入/更新/刪除數據

這是一個鏈接

部署應用程序時，您需要將庫添加到您的 jar