How can I setup a distributed cache for Pants?

Question

From the documentation site, Pants mentions in the First Concepts that it supports the notion of a distributed cache to cache published artifacts. See https://pantsbuild.github.io/first_concepts.html .

I have been looking around in the documentation for steps to setup a distributed cache, but have been unsuccessful. Can someone point me in the right direction for instructions to follow?

Answer 1

首先查看是否有帮助： https ： //pantsbuild.github.io/setup_repo.html#outside-caches并进行报告。

Answer 2

First of all, you want to make your pants setup so that you are getting "good" values written to cache. Pants will not upload the results of incremental builds to the cache, so what we usually recommend is that you setup your CI environment to publish to the cache, and then set these pants.ini settings to keep zinc from performing incremental builds in CI.

[compile.zinc]
# We don't want to use incremental compile in CI.  This should short-circuit
# some extra work Pants does to support incremental compile.
incremental: False

# Introduce some randomness when choosing the next chunk of work in order
# to share more work between different CI jobs.
size_estimator: random

There are two setups in common use: a REST cache and an NFS cache

It can be convenient to use NFS if all machines are able to access a common NFS server. The configuration for the CI workers is pretty simple:

[cache]
read_from: ["/mnt/nfs/shared-cache/pants-artifact-cache"]
read: True
write_to: ["/mnt/nfs/shared-cache/pants-artifact-cache"]
# For CI builds, turn on writing, but for developer workstations you may
# want to leave this off and mount the NFS dir read-only
write: True
# Turn off auto-purging of the buildcache, leave that to a cron job
max_entries_per_target: 0

Here's the cleanup job installed as a cronjob:

echo /usr/sbin/tmpwatch -c 14d /data2/shared-cache/pants-artifact-cache > /etc/cron.daily/pants-cache-clean
chmod +x /etc/cron.daily/pants-cache-clean

If you want to setup a cache that is shared using REST, you can configure a server to accept PUT and DELETE calls. Then configure the cache settings with URLs as follows:

[cache]
read_from: ["https://pantscache.example.com/pants-artifact-cache/ro"]
read: True
write_to: ["https://pantscache.example.com/pants-artifact-cache/rw"]
# For CI builds, turn on writing, but for developer workstations you may
# want to leave this off.
write: True    

See this writeup for setting up a basic NGINX with Pants. Ask on the pants-devel@ group or pantsbuild #general slack group if you want details on more sophisticated cache setups using Varnish or multiple NGINX servers with sharding.