GitLab Pages CI cache appears to cache and restore nothing. How do I inspect the cache?

Question

I have a .gitlab-ci.yml that looks like this:

image: mcr.microsoft.com/playwright:latest

before_script:
- apt-get remove nodejs -y
- apt-get update
- apt-get install -y curl
- curl -fsSL https://deb.nodesource.com/setup_18.x | bash -
- apt-get install -y nodejs
- npm install

pages:
  stage: build
  script:
  - npm run docsify ; npx next build ; npx next export
  - rm -rf public
  - mv out public
  artifacts:
    paths:
    - public
  cache:
    key: default
    paths:
    - node_modules/
    - ".next/cache/"
    - public
    untracked: true # unclear if needed
  only:
  - main

And it runs and it looks like it does all the right things:

Running with gitlab-runner 14.9.1 (bd40e3da)
  on runner-gitlab-runner-7ff6fc5d7b-dc9sh 2bfx5V6B
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image mcr.microsoft.com/playwright:latest ...
Using attach strategy to execute scripts...
Preparing environment
Waiting for pod gitlab-managed-apps/runner-2bfx5v6b-project-3385-concurrent-06khk6 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-2bfx5v6b-project-3385-concurrent-06khk6 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-2bfx5v6b-project-3385-concurrent-06khk6 via runner-gitlab-runner-7ff6fc5d7b-dc9sh...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in 
Created fresh repository.
Checking out 456ffc27 as main...
Skipping Git submodules setup
Restoring cache
Checking cache for default-4...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script

...

Saving cache for successful job
Creating cache default-4...
node_modules/: found 22078 matching files and directories 
.next/cache/: found 18 matching files and directories 
public: found 925 matching files and directories   
untracked: found 21814 files                       
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. 
Created cache

So it should be caching:

• node_modules/
• .next/cache/
• public

My understanding is that after the next git clone of the next pipeline run, GitLab CI should restore the cache on top of the cloned repository. (Or make the cache available via a volume mount, but I haven't been able to find it in /cache or ./cache .)

Printing out the contents of public in the next commit reveals this is not the case.

But by all appearances the cache gets restored:

Fetching changes with git depth set to 50...
Initialized empty Git repository in 
Created fresh repository.
Checking out d7d84cd8 as main...
Skipping Git submodules setup
Restoring cache
Checking cache for default-4...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache

Where is the cache supposed to be?

How am I supposed to inspect it to see if it's working?

---

Addendum:

From a Next.js standpoint, this message would indicate that Next.js is not benefiting from the cache either:

warn  - No build cache found. Please configure build caching for faster rebuilds. Read more: https://nextjs.org/docs/messages/no-cache

And I have the lines that they suggest you merge into your .gitlab-ci.yml to take advantage of caching.

Answer 1

You haven't configured distributed caching, so your caches are only stored locally on the particular runner that runs the job when the cache is uploaded.

Caches are not shared between runners. So, if you have multiple runners, the cache may not be available if your job is picked up by a different runner than last time. This is because GitLab assumes different runners can potentially be on different platforms and architectures and cache files may depend on these environmental factors (for binary compatibility, for example). If you have multiple runners or are using autoscaling features, you will need to configure distributed caching .

If you only have a single runner but still have this issue, make sure you don't have a cache mismatching problem with your configuration.

The log messages in your jobs indicate the cache is being created and restored (from local cache) successfully.

Is S3 required for shared runners?

Distributed caching is required if you have multiple runners or if you are using an autoscaling configuration as mentioned above. However, you don't necessarily have to use AWS S3 for this, just S3- compatible object storage. For example, you can self-host minio which provides (among other things) an S3-compatible object storage server.

Or make the cache available via a volume mount, but I haven't been able to find it in /cache or ./cache.

This is from the runner pod perspective. The runner will extract the compressed cache (if it exists) from /cache (mounted to the runner pod) to the build directory automatically. Jobs will not access the /cache mount directly! However, it's good to know this if you want to manually inspect the cache on your runner pod.

You can inspect the cache by logging into a shell on your runner pod (using kubectl exec -it <your-runner-pod-name> /bin/sh ) and inspecting the /cache directory and any compressed archives in it.

Answer 2

确保您的跑步者配置为使用分布式缓存 - https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching

Answer 3

If you want to use caching, you have to set up distributed caching.

If you want to set up distributed caching, you have to setup an S3-compatible object storage.

There is no out-of-the-box support for caching without object storage.

Even if you use the same runner for every build, it does not persist.