change log files output path for flink jobs that run on yarn

Question

We have a few flink jobs that run on yarn. We would like to upload flink job logs to ELK to simplify debugging/analysis. Currently flink task managers write logs to /mnt/flinklogs/$application_id/$container_id . We want to have it write to a directory without $applicatoin_id/$container_id nested structure.

I tried with env.log.dir: /mnt/flink . With this setting, the configuration is not passed correctly.

 -Dlog.file=/mnt/flinklogs/application_1560449379756_1312/\
    container_e02_1560449379756_1312_01_000619/taskmanager.log

Answer 1

我认为解决此问题的最佳方法是使用纱线日志聚合将日志写入磁盘，并使用弹性文件位将其发送到弹性。