How do you hide sensitive configuration data from the Apache Flink configuration dashboard?

Question

We're starting to use Apache Flink for data streaming. It's processing data and then updating the database in the sinks. We configure the database endpoints/credentials as Flink Configuration data (as we have the standard dev/test/prod setup), but unfortunately the passwords/api-keys are showing in plaintext in the Flink Configuration dashboard, along with all the other configuration data.

Is there a way of configuring flink so that the java code can still access the sensitive password information via ParameterTool (or similar) but the sensitive data is not displayed in the Flink Dashboard?

Answer 1

For the cluster level configuration ( flink-conf.yaml ) this feature is supported. What one needs to do in order to not display secret values is to include secret or password in the key (eg the values of cluster.password or secret.plan won't be displayed).

However, this feature is not yet supported for the user configuration as you would need it when using the ParameterTool and passing global job parameters via ExecutionConfig.setGlobalJobParameters . I've filed FLINK-14047 in order to add this feature to Flink.

Update

FLINK-14047 has been merged into Flink 1.10. Now also sensitive user configuration values will be filtered out if their key names contain either password or secret .