How to ignore spark-submit warnings for pyspark

Question

When I submit my python file to spark like this

spark-submit driver.py

It starts showing a lot of warning related to python 2 print method.

18/10/19 01:37:52 WARN ScriptBasedMapping: Exception running /etc/hadoop/conf/topology_script.py 10.5.1.112
ExitCodeException exitCode=1:   File "/etc/hadoop/conf/topology_script.py", line 63
    print rack
             ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(rack)?

Is there any way like spark-submit driver.py -W ignore ??

>> I know this is the warning of python2, my code is in python3 so I just want to ignore the python2 warnings. These warnings start showing even before control goes to my code. Because spark-submit runs first and later on load python files.

Answer 1

You're using python 3, but writing in Python 2?

Please refer to this question : What does "SyntaxError: Missing parentheses in call to 'print'" mean in Python?

You can also refer to Configure logging in Apache Spark to set the default loglevel using a log4j.properties file.

You can also set the loglevel when you call spark-submit or inside the code using sc.setLogLevel("WARN")

Just another SO question for reference : How to stop INFO messages displaying on spark console?

Answer 2

If you want to ignore a Python warning, maybe this could work:

import warnings
warnings.simplefilter(action='ignore', category=YourWarningCategory)