Segregating log files for jobs in a shared flink cluster

Question

Not able to run in kube at the moment so using a shared cluster. However the logs for all jobs become comingled in the TM. Is there any way to label the log lines so that that can be distinguished and tied back to the individual jobs? Eg something with an NDC or something more explicit in the logging setup?

Cheers.

Answer 1

It wouldn't always be meaningful to try to attribute log lines to just one job, but regardless, the answer is no, there's no support for labeling the TM's log lines to indicate which job is involved.

Answer 2

I don't know what aggregation tooling you are using but Loki automatically adds source filename tag - you can use that to distinguish TMs. Your tool might have something like that as well.