[英]How to ignore spark-submit warnings for pyspark
When I submit my python file to spark like this 当我提交我的python文件以这种方式触发时
spark-submit driver.py
It starts showing a lot of warning related to python 2 print method. 它开始显示很多与python 2打印方法有关的警告。
18/10/19 01:37:52 WARN ScriptBasedMapping: Exception running /etc/hadoop/conf/topology_script.py 10.5.1.112
ExitCodeException exitCode=1: File "/etc/hadoop/conf/topology_script.py", line 63
print rack
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(rack)?
Is there any way like spark-submit driver.py -W ignore
?? 有没有办法像
spark-submit driver.py -W ignore
??
>> I know this is the warning of python2, my code is in python3 so I just want to ignore the python2 warnings. >>我知道这是python2的警告,我的代码在python3中,所以我只想忽略python2警告。 These warnings start showing even before control goes to my code.
这些警告甚至在控制权进入我的代码之前就开始显示。 Because
spark-submit
runs first and later on load python files. 因为
spark-submit
首先在加载python文件上运行,然后再运行。
You're using python 3, but writing in Python 2? 您使用的是python 3,但使用python 2编写?
Please refer to this question : What does "SyntaxError: Missing parentheses in call to 'print'" mean in Python? 请参考此问题: “ SyntaxError:在调用'print'时缺少括号”在Python中意味着什么?
You can also refer to Configure logging in Apache Spark to set the default loglevel using a log4j.properties file. 您还可以参考在Apache Spark中配置日志记录以使用log4j.properties文件设置默认日志级别。
You can also set the loglevel when you call spark-submit
or inside the code using sc.setLogLevel("WARN")
您还可以在调用
spark-submit
或在代码内部使用sc.setLogLevel("WARN")
Just another SO question for reference : How to stop INFO messages displaying on spark console? 另一个SO问题供参考: 如何停止INFO消息在spark控制台上显示?
If you want to ignore a Python warning, maybe this could work: 如果您想忽略Python警告,也许可以这样做:
import warnings
warnings.simplefilter(action='ignore', category=YourWarningCategory)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.