Selectively deploying artifacts built by Bazel

Question

I generate artifacts via bazel build . I want to deploy these artifacts only when they change .

What's the best way to detect whether an artifact changed?

...is there a way to get the digest for a build target? Ideally without needing to build it?

Answer 1

The approach here depends a lot on all the details.

One way is to set up something using the Build Event Protocol: https://docs.bazel.build/versions/master/build-event-protocol.html

That might be more setup than you want to deal with.

Another approach is to use genrules or similar to hash all the artifacts you're interested in (I'm not aware of a way to get bazel to output its hashes of artifacts). Then create a binary ( sh_binary , or something) that takes as input all those artifacts you're interested in and their hashes, and compares those hashes against the hashes of what's deployed, and then deploys what's changed.

Another approach is to create a Starlark rule that registers a "deploy action" for each thing you're interested in deploying. Bazel will run the deploy actions for only those artifacts that have changed each time you run the deploy target. That might look something like this:

defs.bzl :

def _deploy_impl(ctx):

  deploy_results = []

  for deployable in ctx.attr.deployables:
    deploy_result = ctx.actions.declare_file(
        ctx.label.name + "_" + deployable.label.name + "_deploy_result")
    deploy_results.append(deploy_result)

    deploy_args = ctx.actions.args()
    deploy_args.add(deploy_result)
    deploy_args.add(ctx.attr.destination)
    deploy_args.add_all(deployable.files)

    ctx.actions.run(
        executable = ctx.attr._deploy_tool.files_to_run,
        inputs = deployable.files,
        outputs = [deploy_result],
        arguments = [deploy_args],
        execution_requirements = {
          "no-sandbox": "1",
          "local": "1",
          "no-cache": "1",
          "no-remote-cache": "1",
          "no-remote": "1",
        })

  return [DefaultInfo(files = depset(deploy_results))]

deploy = rule(
  implementation = _deploy_impl,
  attrs = {
    "deployables": attr.label_list(),
    "destination": attr.string(mandatory=True),
    "_deploy_tool": attr.label(default = "//:deploy_tool", cfg = "host"),
  }
)

deploy_tool.sh :

result_file_path="$1"
shift
destination="$1"
shift

echo "deploying $@ to $destination"
cp -f --target-directory "$destination" $@

touch "$result_file_path"

BUILD :

load(":defs.bzl", "deploy")

genrule(
  name = "gen_a",
  srcs = ["a"],
  outs = ["a.out"],
  cmd = "cp $< $@",
)

genrule(
  name = "gen_b",
  srcs = ["b"],
  outs = ["b.out"],
  cmd = "cp $< $@",
)

genrule(
  name = "gen_c",
  srcs = ["c"],
  outs = ["c.out"],
  cmd = "cp $< $@",
)

deploy(
  name = "deploy",
  deployables = [
    ":gen_a",
    ":gen_b",
    ":gen_c",
  ],
  destination = "/tmp",
)

sh_binary(
  name = "deploy_tool",
  srcs = ["deploy_tool.sh"],
)

usage:

$ bazel build deploy
INFO: Analyzed target //:deploy (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: From Action deploy_gen_a_deploy_result:
deploying bazel-out/k8-fastbuild/bin/a.out to /tmp
INFO: From Action deploy_gen_c_deploy_result:
deploying bazel-out/k8-fastbuild/bin/c.out to /tmp
INFO: From Action deploy_gen_b_deploy_result:
deploying bazel-out/k8-fastbuild/bin/b.out to /tmp
Target //:deploy up-to-date:
  bazel-bin/deploy_gen_a_deploy_result
  bazel-bin/deploy_gen_b_deploy_result
  bazel-bin/deploy_gen_c_deploy_result
INFO: Elapsed time: 0.146s, Critical Path: 0.02s
INFO: 3 processes: 3 local.
INFO: Build completed successfully, 5 total actions

Some notes:

1. The "deploy actions" are very non-hermetic by their nature, which is generally considered not good, but here it's fine. In particular, look at all the "execution_requirements" for the action.

2. Bazel will try to run all the deploy actions in parallel, which might be good or bad.

3. The deploy rule could be made to be an executable rule so that it can be used with bazel run , which might sound nicer.

4. This example could probably also be done using macros, if you really wanted, but it would create a lot of "artificial" targets.