[英]How to turn on TRACE logging in spark
我注意到,每次催化劑改變計划時,spark中的RuleExecutor都會執行跟蹤日志:
我想知道的是如何配置spark以便打開跟蹤日志記錄? 我正在使用log4j,並且遇到了以下文檔: https : //jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-logging.html
我一直在研究代碼一段時間,並且看到可以將'log4j.threshold = TRACE'設置為使記錄器的一部分處於跟蹤模式,但是我似乎無法使催化劑使用催化劑拿起設置。
我究竟做錯了什么?
我只是嘗試了一個簡單的結構化流程序,從IntelliJ中的Kafka讀取數據,以下語句對我有用,即給了我跟蹤級別的日志:
SparkSession.builder().getOrCreate().sparkContext().setLogLevel("TRACE");
這是顯示某些跟蹤日志的輸出的一部分:
...
...
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
+-----+----+-----+-----------------------+
|topic|key |value|timestamp |
+-----+----+-----+-----------------------+
|test |null|hi345|2018-10-12 23:56:00.099|
+-----+----+-----+-----------------------+
18/10/12 23:56:02 DEBUG GenerateUnsafeProjection: code for input[0, string, true],input[1, string, true],input[2, string, true],input[3, string, true]:
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */ return new SpecificUnsafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
/* 006 */
/* 007 */ private Object[] references;
...
...
希望這可以幫助!
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.