EMR 6.9 中的 EMR 6.7 配置给出错误 Classification 'spark-log4j' is not valid for parent classification 'null'

Question

I was using emr 6.7 with the software configuration:

[
  {
    "Classification": "spark",
    "Properties": {
      "maximizeResourceAllocation": "true"
    }
  },
  {
    "Classification": "spark-log4j",
    "Properties": {
      "log4j.rootCategory": "ERROR, console"
    }
  }
]

but for some reason when I shifted to emr 6.9.

The was website started throwing error

Classification 'spark-log4j' is not valid for parent classification 'null'.

If I remove this spark-log4j then it works but starts giving unnecessary INFO and DEBUG logs

How can I configure spark-log4j in EMR 6.9?

Answer 1

There is a classification of 'spark-log4j2' now: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-690-release.html#emr-690-class . I guess using that would solve your problem.