How to set pyspark logging level to debug?
Question
Can anyone help me with the spark configuration needed to set logging level to debug and capture more logs
1 answers
solution1
0 ACCPTED 2020-05-06 18:19:53
Set setLogLevel property to DEBUG in sparksession.
from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('test').getOrCreate()
spark.sparkContext.setLogLevel("DEBUG")
#or
log4j = sc._jvm.org.apache.log4j
log4j.LogManager.getRootLogger().setLevel(log4j.Level.DEBUG)
#for custom log
log4j.LogManager.getRootLogger().info("custom log")
#2020-05-06 13:43:42 INFO root:? - custom log
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Logging not displaying debug level when level is set by dictConfig
How to set logging level in python?
How to set logging level in twisted?
How to set logging level names?
how to logging debug level to file but only info level on screen in python
How to set logging level for the module only in Python?
how to set logging level from command line
How to set logging level in a unit test for Python
How to set Logging Level from getLogger instance
Logging: how to set a maximum log level for a handler
Related Tags