Spark 结构化流 UI 的自定义选项卡

Question

I have created a Custom Streaming tab for Spark Structured streaming. I have just attached the streaming Tab in Spark UI but i am not able to attach page in streaming tab. In this custom tab, i want to show how many batches has completed, number of messages coming per batch, Processing Time and its graph. Through SQL listener and StreamingQueryListener i can get the information and want to add in streaming tab.

public class CustomTab extends SparkUITab {

public CustomTab(SparkUI parent, String prefix) {
    super(parent, prefix);
}

public static void getCustomTab(JobContext jobcontext) {
    SparkContext sparkContext = jobcontext.getSparkSession().sparkContext();

    StreamingContext ssc = new StreamingContext(sparkContext, new Duration(1000));
    StreamingTab streamingTab = new StreamingTab(ssc);

    sparkContext.ui().get().attachPage(new CustomPage("customPage", streamingTab));

    if (sparkContext.ui().isDefined()) {
        sparkContext.ui().get().attachTab(streamingTab);

    } 
}

}

I am getting issue while attaching customPage in custom streaming tab. render method of SparkUITab is not called by SparkUI. Second issue I am getting is that how to change scala.xml.Node class in Java. I mean is there any class in java that I can use instead of Node class or how to add String in scala.xml.Node.

If there is any Java class that I can use, so how to change back to Seq of Node.

class CustomPage extends WebUIPage {

StreamingTab streamingTab = null;

public CustomPage(String prefix, StreamingTab sparkUI) {
    super(prefix);
    this.streamingTab = sparkUI;
}

@Override
public Seq<Node> render(HttpServletRequest request) {
    System.out.println("In render method");

    String value = "This text is going to come at the bottom";
    Option<String> optionSome = Option.apply(value);

    Function0<Seq<Node>> length = new AbstractFunction0<Seq<Node>>() {

        @Override
        public Seq<Node> apply() {
            scala.xml.Node node = null;
        //  I am not getting that how to add html code("<div> { <div id=\"custommmmmm\"></div> }  </div>") here
            return null;
        }

    };

    return UIUtils.headerSparkPage("custom", length, this.streamingTab, null, optionSome, false, false);
}

}

Is it possible to show structured streaming batches, its processing time and no of rows per batch through listeners and if possible then which listener should i use and how to add that in my custom page.

Answer 1

You can use StreamingQueryListener for your use case. You'll have to persist the data you get from StreamingQueryListener and then you can display them in your Custom tab.

Answer 2

Use the class Text - see the example:

UIUtils.headerSparkPage(request,
                "My page",
                () -> new Text("my custom page content"), parent, Option.apply("help text"),false, false);