EMR - Airflow 运行 scala jar 文件 airflow.exceptions.AirflowException

Question

我正在尝试使用 emr 从 AIRFLOW 运行 scala jar 文件，而 jar 文件旨在读取 mssql-jdbc 和 postgresql。

从 airflow，我能够创建集群

我的 SPARK_STEPS 看起来像

SPARK_STEPS = [
    {
        'Name': 'Trigger_Source_Target',
        'ActionOnFailure': 'CONTINUE',
        'HadoopJarStep': {
            'Jar': 'command-runner.jar',
            'Args': ['spark-submit', 
                     '--master', 'yarn', 
                     '--jars', '/mnt/MyScalaImport.jar', 
                     '--class', 'org.classname', 
                     's3://path/SNAPSHOT.jar', 
                     'SQL_Pwd', 'PostgreSQL_PWD', 'loadtype'],
        }
    }
]

在此之后我定义了 JOB_FLOW_OVERRIDES-

JOB_FLOW_OVERRIDES = {
    "Name": "pfdt-cluster-airflow",
    "LogUri": "s3://path/elasticmapreduce/",
    "ReleaseLabel": "emr-6.4.0",
    "Applications": [
        {"Name": "Spark"},
    ],
    "Instances": {
        "InstanceGroups": [
            {
                "Name": "Master nodes",
                "Market": "ON_DEMAND",
                "InstanceRole": "MASTER",
                "InstanceType": "m5.xlarge",
                "InstanceCount": 1,
            }
        ],
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        'Ec2KeyName': 'pem_file_name',
        "Ec2SubnetId": "subnet-123"
    },
    'BootstrapActions': [
        {
            'Name': 'import custom Jars',
            'ScriptBootstrapAction': {
                'Path': 's3://path/subpath/copytoolsjar.sh',
                'Args': []
            }
        }
    ],
    'Configurations': [
      {
        'Classification': 'spark-defaults',
        'Properties': {
          'spark.jars': 's3://jar_path/mssql-jdbc-8.4.1.jre8.jar'
         }
      }
    ],
    "VisibleToAllUsers": True,
    "JobFlowRole": "EMR_EC2_DefaultRole",
    "ServiceRole": "EMR_DefaultRole",
    "Tags": [
        {"Key": "Environment", "Value": "Development"},
    ],
}

要将 scala.jar 文件从 S3 复制到本地到气流-我有一个 shell 脚本可以完成工作：Path-s3://path/subpath/copytoolsjar.sh

aws s3 cp s3://path/SNAPSHOT.jar /mnt/MyScalaImport.jar

在触发气流时 - 它在节点 watch_step 失败

我得到的错误是- stdout.gz => 标准错误.gz =>

22/04/08 13:38:23 INFO CodeGenerator: Code generated in 25.5907 ms Exception in thread "main" java.sql.SQLException: No suitable driver at java.sql.DriverManager.getDriver(DriverManager.java:315) at org.apache.spark .sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$2(JDBCOptions.scala:108) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.execution.datasources.jdbc .JDBCOptions.(JDBCOptions.scala:108) 在 org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.(JDBCOptions.scala:38)

如何解决这个问题 - 我有我的 jars 在 -

s3://path/subpath/mssql-jdbc-8.4.1.jre8.jar

s3://路径/子路径/postgresql-42.2.24.jar

Answer 1

上传jar文件（mssql-jdbc-8.4.1.jre8.jar，postgresql-42.2.24.jar）到airflow local-

在引导步骤'BootstrapActions': [ { 'Name': 'import custom Jars', 'ScriptBootstrapAction': { 'Path': 's3://path/subpath/copytoolsjar.sh', 'Args': [] } } ]

在 copytoolsjar.sh 文件中写入命令aws s3 cp cp s3://path/SNAPSHOT.jar /mnt/MyScalaImport.jar && bash -c "sudo aws s3 cp s3://path/subpath/mssql-jdbc-8.4.1.jre8.jar /usr/lib/spark/jars/" && bash -c "sudo aws s3 cp s3://path/subpath/postgresql-42.2.24.jar /usr/lib/spark/jars/"工作将会完成