Where are Staged files downloaded to on GCP Dataflow Workers

Question

I am working on using GCP Dataflow Flex Templates using Java. I want to stage '/usr/local/jdk-11-latest/lib/security/cacerts' and '/usr/local/jdk-11-latest/lib/security/keystore.jks' for TLS validation. I have checked that the files are staged for the dataflow runners, but they are not downloaded to the same paths as they were used to upload. Where does GCP Dataflow put staged files it downloads from GCS?

Answer 1

If you open the cloud logs of your job and search in dataflow.googleapis.com/worker-startup logs, you can see logging that looks like Downloading: gs://.../xxx.jar to /var/opt/google/tmp/download.0.219961590/file.0 . Then when you search the java cmd logging, you can see all the jars are located under /var/opt/google/dataflow/ .