How do you enumerate and copy multiple files to the source folder in Bazel?

Question

How do you enumerate and copy multiple files to the source folder in Bazel?

I'm new to Bazel and I am trying to replace a non-Bazel build step that is effectively cp -R with an idiomatic Bazel solution. Concrete use cases are:

• copying.proto files to aa sub-project where they will be picked up by a non-Bazel build system. There are N .proto files in N Bazel packages, all in one protos/ directory of the repository.
• copying numerous .gotmpl template files to a different folder where they can be picked up in a docker volume for a local docker-compose development environment. There are M template files in one Bazel package in a small folder hierarchy. Example code below.
• Copy those same .gotmpl files to a gitops-type repo for a remote terraform to send to prod.

All sources are regular, checked in files in places where Bazel can enumerate them. All target directories are also Bazel packages. I want to write to the source folder, not just to bazel-bin , so other non-Bazel tools can see the output files.

Currently when adding a template file or a proto package, a script must be run outside of bazel to pick up that new file and add it to a generated.bzl file, or perform operations completely outside of Bazel. I would like to eliminate this step to move closer to having one true build command.

I could accomplish this with symlinks but it still has an error-prone manual step for the.proto files and it would be nice to gain the option to manipulate the files programmatically in Bazel in the build.

Some solutions I've looked into and hit dead ends:

• glob seems to be relative to current package and I don't see how it can be exported since it needs to be called from BUILD. A filegroup solves the export issue but doesn't seem to allow enumeration of the underlying files in a way that a bazel run target can take as input.
• Rules like cc_library that happily input globs as srcs are built into the Bazel source code, not written in Starlark
• genquery and aspects seem to have powerful meta-capabilities but I can't see how to actually accomplish this task with them.
• The "bazel can write to the source folder" pattern and write_source_files from aspect-build/bazel-lib might be great if I could programmatically generate the files parameter.

Here is the template example which is the simpler case. This was my latest experiment to bazel-ify cp -R . I want to express src/templates/update_templates_bzl.py in Bazel.

src/templates/BUILD :

# [...]
exports_files(glob(["**/*.gotmpl"]))
# [...]

src/templates/update_templates_bzl.py :

#!/usr/bin/env python

from pathlib import Path

parent = Path(__file__).parent
template_files = [str(f.relative_to(parent)) for f in list(parent.glob('**/*.gotmpl'))]
as_python = repr(template_files).replace(",", ",\n    ")

target_bzl = Path(__file__).parent / "templates.bzl"

target_bzl.write_text(f""""Generated template list from {Path(__file__).relative_to(parent)}"

TEMPLATES = {as_python}""")

src/templates/copy_templates.bzl

"""Utility for working with this list of template files"""

load("@aspect_bazel_lib//lib:write_source_files.bzl", "write_source_files")
load("templates.bzl", "TEMPLATES")

def copy_templates(name, prefix):
    files = {
        "%s/%s" % (prefix, f) : "//src/templates:%s" % f for f in TEMPLATES
    }

    write_source_files(
        name = name,
        files = files,
        visibility = ["//visibility:public"],
    )

other/module :

load("//src/templates:copy_templates.bzl", "copy_templates")

copy_templates(
    name = "write_template_files",
    prefix = "path/to/gitops/repo/templates",
)

Answer 1

One possible method to do this would be to use google/bazel_rules_install .

As mentioned in the project README.md you need to add the following to your WORKSPACE file;

# file: WORKSPACE
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_github_google_rules_install",
    urls = ["https://github.com/google/bazel_rules_install/releases/download/0.3/bazel_rules_install-0.3.tar.gz"],
    sha256 = "ea2a9f94fed090859589ac851af3a1c6034c5f333804f044f8f094257c33bdb3",
    strip_prefix = "bazel_rules_install-0.3",
)

load("@com_github_google_rules_install//:deps.bzl", "install_rules_dependencies")

install_rules_dependencies()

load("@com_github_google_rules_install//:setup.bzl", "install_rules_setup")

install_rules_setup()

Then in your src/templates directory you can add the following to bundle all your templates into one target.

# file: src/templates/BUILD.bazel
load("@com_github_google_rules_install//installer:def.bzl", "installer")

installer(
    name = "install_templates",
    data = glob(["**/*.gotmpl"]),
)

Then you can use the installer to install into your chosen directory like so.

bazel run //src/templates:install_templates -- path/to/gitops/repo/templates

It's also worth checking out bazelbuild/rules_docker for building your development environments using only Bazel.