[英]Gitlab CI gradle dependency cache
I'm building android on Gitlab CI and downloading dependencies each time is annoying.我正在 Gitlab CI 上构建 android 并且每次下载依赖项都很烦人。 I tried caching:
我尝试缓存:
$HOME/.gradle/
$HOME/.gradle/caches/
$GRADLE_HOME/caches/
$HOME/.m2/
.gradle/
build/
app/build/
The 26594th iteration of my.gitlab-ci.yml is here . my.gitlab-ci.yml 的第 26594 次迭代在这里。 I've installed gradle in my build environment because without caching gradlew downloads it every time, which takes even more time.
我已经在我的构建环境中安装了 gradle 因为没有缓存 gradlew 每次都会下载它,这需要更多时间。 The image I use for building ishere .
我用于构建的图像在这里。 My CI server startup script can be found here .
我的 CI 服务器启动脚本可以在这里找到。 I assume that, if caching works, I won't see a list of my dependencies downloading on every build.
我假设,如果缓存有效,我不会在每个构建中看到我的依赖项列表下载。
Build log:构建日志:
gitlab-ci-multi-runner 0.7.2 (998cf5d) Using Docker executor with image android... Pulling docker image gitlab/gitlab-runner:build... Pulling docker image android:latest... WARNING: Cannot pull the latest version of image android:latest: Error: image library/android:latest not found WARNING: Locally found image will be used instead. gitlab-ci-multi-runner 0.7.2 (998cf5d) Using Docker executor with image android... Pulling docker image gitlab/gitlab-runner:build... Pulling docker image android:latest... WARNING: Cannot pull the latest图像版本 android:latest: 错误: image library/android:latest not found 警告: 将使用本地找到的图像。
Running on runner-76acd9d3-project-74192-concurrent-0 via ff9200d4be41...
Fetching changes...
Removing .gradle/
Removing artifacts.upload.log
Removing rxfirebaseandroid/build/
HEAD is now at 7f151df djgfj
Checking out 7f151df2 as ci-gitlab...
HEAD is now at 7f151df... djgfj
Restoring cache...
$ gradle testDebug
Download https://jcenter.bintray.com/com/android/tools/build/gradle/1.3.0/gradle-1.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/build/gradle-core/1.3.0/gradle-core-1.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/build/builder/1.3.0/builder-1.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/lint/lint/24.3.0/lint-24.3.0.pom
Download https://jcenter.bintray.com/net/sf/proguard/proguard-gradle/5.2.1/proguard-gradle-5.2.1.pom
Download https://jcenter.bintray.com/net/sf/proguard/proguard-parent/5.2.1/proguard-parent-5.2.1.pom
Download https://jcenter.bintray.com/com/android/tools/build/builder-model/1.3.0/builder-model-1.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/build/builder-test-api/1.3.0/builder-test-api-1.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/sdklib/24.3.0/sdklib-24.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/sdk-common/24.3.0/sdk-common-24.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/common/24.3.0/common-24.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/build/manifest-merger/24.3.0/manifest-merger-24.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/ddms/ddmlib/24.3.0/ddmlib-24.3.0.pom
Download https://jcenter.bintray.com/com/squareup/javawriter/2.5.0/javawriter-2.5.0.pom
Download https://jcenter.bintray.com/org/sonatype/oss/oss-parent/7/oss-parent-7.pom
Download https://jcenter.bintray.com/org/bouncycastle/bcpkix-jdk15on/1.48/bcpkix-jdk15on-1.48.pom
Download https://jcenter.bintray.com/org/bouncycastle/bcprov-jdk15on/1.48/bcprov-jdk15on-1.48.pom
Download https://jcenter.bintray.com/org/ow2/asm/asm/5.0.3/asm-5.0.3.pom
Download https://jcenter.bintray.com/org/ow2/asm/asm-parent/5.0.3/asm-parent-5.0.3.pom
Download https://jcenter.bintray.com/org/ow2/ow2/1.3/ow2-1.3.pom
Download https://jcenter.bintray.com/org/ow2/asm/asm-tree/5.0.3/asm-tree-5.0.3.pom
Download https://jcenter.bintray.com/com/android/tools/jack/jack-api/0.9.0/jack-api-0.9.0.pom
Download https://jcenter.bintray.com/com/android/tools/jill/jill-api/0.9.0/jill-api-0.9.0.pom
Download https://jcenter.bintray.com/com/android/tools/lint/lint-checks/24.3.0/lint-checks-24.3.0.pom
Download https://jcenter.bintray.com/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.pom
Download https://jcenter.bintray.com/net/sf/proguard/proguard-base/5.2.1/proguard-base-5.2.1.pom
Download https://jcenter.bintray.com/com/android/tools/annotations/24.3.0/annotations-24.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/layoutlib/layoutlib-api/24.3.0/layoutlib-api-24.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/dvlib/24.3.0/dvlib-24.3.0.pom
Download https://jcenter.bintray.com/com/google/code/gson/gson/2.2.4/gson-2.2.4.pom
Download https://jcenter.bintray.com/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.pom
Download https://jcenter.bintray.com/org/apache/commons/commons-parent/33/commons-parent-33.pom
Download https://jcenter.bintray.com/org/apache/apache/13/apache-13.pom
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.1.1/httpclient-4.1.1.pom
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-client/4.1.1/httpcomponents-client-4.1.1.pom
Download https://jcenter.bintray.com/org/apache/httpcomponents/project/4.1.1/project-4.1.1.pom
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpmime/4.1/httpmime-4.1.pom
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-client/4.1/httpcomponents-client-4.1.pom
Download https://jcenter.bintray.com/com/google/guava/guava/17.0/guava-17.0.pom
Download https://jcenter.bintray.com/com/google/guava/guava-parent/17.0/guava-parent-17.0.pom
Download https://jcenter.bintray.com/net/sf/kxml/kxml2/2.3.0/kxml2-2.3.0.pom
Download https://jcenter.bintray.com/com/android/tools/lint/lint-api/24.3.0/lint-api-24.3.0.pom
Download https://jcenter.bintray.com/org/ow2/asm/asm-analysis/5.0.3/asm-analysis-5.0.3.pom
Download https://jcenter.bintray.com/com/intellij/annotations/12.0/annotations-12.0.pom
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.1/httpcore-4.1.pom
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-core/4.1/httpcomponents-core-4.1.pom
Download https://jcenter.bintray.com/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.pom
Download https://jcenter.bintray.com/org/apache/commons/commons-parent/5/commons-parent-5.pom
Download https://jcenter.bintray.com/org/apache/apache/4/apache-4.pom
Download https://jcenter.bintray.com/commons-codec/commons-codec/1.4/commons-codec-1.4.pom
Download https://jcenter.bintray.com/org/apache/commons/commons-parent/11/commons-parent-11.pom
Download https://jcenter.bintray.com/com/android/tools/external/lombok/lombok-ast/0.2.3/lombok-ast-0.2.3.pom
Download https://jcenter.bintray.com/com/android/tools/build/gradle/1.3.0/gradle-1.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/build/gradle-core/1.3.0/gradle-core-1.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/build/builder/1.3.0/builder-1.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/lint/lint/24.3.0/lint-24.3.0.jar
Download https://jcenter.bintray.com/net/sf/proguard/proguard-gradle/5.2.1/proguard-gradle-5.2.1.jar
Download https://jcenter.bintray.com/com/android/tools/build/builder-model/1.3.0/builder-model-1.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/build/builder-test-api/1.3.0/builder-test-api-1.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/sdklib/24.3.0/sdklib-24.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/sdk-common/24.3.0/sdk-common-24.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/common/24.3.0/common-24.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/build/manifest-merger/24.3.0/manifest-merger-24.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/ddms/ddmlib/24.3.0/ddmlib-24.3.0.jar
Download https://jcenter.bintray.com/com/squareup/javawriter/2.5.0/javawriter-2.5.0.jar
Download https://jcenter.bintray.com/org/bouncycastle/bcpkix-jdk15on/1.48/bcpkix-jdk15on-1.48.jar
Download https://jcenter.bintray.com/org/bouncycastle/bcprov-jdk15on/1.48/bcprov-jdk15on-1.48.jar
Download https://jcenter.bintray.com/org/ow2/asm/asm/5.0.3/asm-5.0.3.jar
Download https://jcenter.bintray.com/org/ow2/asm/asm-tree/5.0.3/asm-tree-5.0.3.jar
Download https://jcenter.bintray.com/com/android/tools/jack/jack-api/0.9.0/jack-api-0.9.0.jar
Download https://jcenter.bintray.com/com/android/tools/jill/jill-api/0.9.0/jill-api-0.9.0.jar
Download https://jcenter.bintray.com/com/android/tools/lint/lint-checks/24.3.0/lint-checks-24.3.0.jar
Download https://jcenter.bintray.com/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.jar
Download https://jcenter.bintray.com/net/sf/proguard/proguard-base/5.2.1/proguard-base-5.2.1.jar
Download https://jcenter.bintray.com/com/android/tools/annotations/24.3.0/annotations-24.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/layoutlib/layoutlib-api/24.3.0/layoutlib-api-24.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/dvlib/24.3.0/dvlib-24.3.0.jar
Download https://jcenter.bintray.com/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar
Download https://jcenter.bintray.com/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.1.1/httpclient-4.1.1.jar
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpmime/4.1/httpmime-4.1.jar
Download https://jcenter.bintray.com/com/google/guava/guava/17.0/guava-17.0.jar
Download https://jcenter.bintray.com/net/sf/kxml/kxml2/2.3.0/kxml2-2.3.0.jar
Download https://jcenter.bintray.com/com/android/tools/lint/lint-api/24.3.0/lint-api-24.3.0.jar
Download https://jcenter.bintray.com/org/ow2/asm/asm-analysis/5.0.3/asm-analysis-5.0.3.jar
Download https://jcenter.bintray.com/com/intellij/annotations/12.0/annotations-12.0.jar
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.1/httpcore-4.1.jar
Download https://jcenter.bintray.com/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
Download https://jcenter.bintray.com/commons-codec/commons-codec/1.4/commons-codec-1.4.jar
Download https://jcenter.bintray.com/com/android/tools/external/lombok/lombok-ast/0.2.3/lombok-ast-0.2.3.jar
Download https://jcenter.bintray.com/com/firebase/firebase-client-android/2.3.1/firebase-client-android-2.3.1.pom
Download https://jcenter.bintray.com/io/reactivex/rxjava/1.0.14/rxjava-1.0.14.pom
Download https://jcenter.bintray.com/com/firebase/firebase-client-jvm/2.3.1/firebase-client-jvm-2.3.1.pom
Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-databind/2.2.2/jackson-databind-2.2.2.pom
Download https://jcenter.bintray.com/com/fasterxml/oss-parent/10/oss-parent-10.pom
Download https://jcenter.bintray.com/com/firebase/tubesock/0.0.11/tubesock-0.0.11.pom
Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-annotations/2.2.2/jackson-annotations-2.2.2.pom
Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-core/2.2.2/jackson-core-2.2.2.pom
Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-annotations/2.2.2/jackson-annotations-2.2.2.jar
Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-core/2.2.2/jackson-core-2.2.2.jar
Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-databind/2.2.2/jackson-databind-2.2.2.jar
Download https://jcenter.bintray.com/com/firebase/tubesock/0.0.11/tubesock-0.0.11.jar
Download https://jcenter.bintray.com/com/firebase/firebase-client-jvm/2.3.1/firebase-client-jvm-2.3.1.jar
Download https://jcenter.bintray.com/com/firebase/firebase-client-android/2.3.1/firebase-client-android-2.3.1.jar
Download https://jcenter.bintray.com/io/reactivex/rxjava/1.0.14/rxjava-1.0.14.jar
Download https://jcenter.bintray.com/junit/junit/4.12/junit-4.12.pom
Download https://jcenter.bintray.com/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom
Download https://jcenter.bintray.com/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom
Download https://jcenter.bintray.com/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
Download https://jcenter.bintray.com/junit/junit/4.12/junit-4.12.jar
:rxfirebaseandroid:preBuild UP-TO-DATE
:rxfirebaseandroid:preDebugBuild UP-TO-DATE
:rxfirebaseandroid:checkDebugManifest
:rxfirebaseandroid:prepareDebugDependencies
:rxfirebaseandroid:compileDebugAidl UP-TO-DATE
:rxfirebaseandroid:compileDebugRenderscript UP-TO-DATE
:rxfirebaseandroid:generateDebugBuildConfig UP-TO-DATE
:rxfirebaseandroid:generateDebugAssets UP-TO-DATE
:rxfirebaseandroid:mergeDebugAssets UP-TO-DATE
:rxfirebaseandroid:generateDebugResValues UP-TO-DATE
:rxfirebaseandroid:generateDebugResources UP-TO-DATE
:rxfirebaseandroid:packageDebugResources UP-TO-DATE
:rxfirebaseandroid:processDebugManifest UP-TO-DATE
:rxfirebaseandroid:processDebugResources UP-TO-DATE
:rxfirebaseandroid:generateDebugSources UP-TO-DATE
:rxfirebaseandroid:processDebugJavaRes UP-TO-DATE
:rxfirebaseandroid:compileDebugJavaWithJavac UP-TO-DATE
:rxfirebaseandroid:preDebugUnitTestBuild UP-TO-DATE
:rxfirebaseandroid:prepareDebugUnitTestDependencies
:rxfirebaseandroid:processDebugUnitTestJavaRes UP-TO-DATE
:rxfirebaseandroid:compileDebugUnitTestJavaWithJavac UP-TO-DATE
:rxfirebaseandroid:compileDebugUnitTestSources UP-TO-DATE
:rxfirebaseandroid:mockableAndroidJar UP-TO-DATE
:rxfirebaseandroid:assembleDebugUnitTest UP-TO-DATE
:rxfirebaseandroid:testDebugUnitTest UP-TO-DATE
BUILD SUCCESSFUL
Total time: 51.086 secs
$ gradle assembleDebug
:rxfirebaseandroid:compileLint
:rxfirebaseandroid:copyDebugLint UP-TO-DATE
:rxfirebaseandroid:preBuild UP-TO-DATE
:rxfirebaseandroid:preDebugBuild UP-TO-DATE
:rxfirebaseandroid:checkDebugManifest
:rxfirebaseandroid:prepareDebugDependencies
:rxfirebaseandroid:compileDebugAidl UP-TO-DATE
:rxfirebaseandroid:compileDebugRenderscript UP-TO-DATE
:rxfirebaseandroid:generateDebugBuildConfig UP-TO-DATE
:rxfirebaseandroid:generateDebugAssets UP-TO-DATE
:rxfirebaseandroid:mergeDebugAssets UP-TO-DATE
:rxfirebaseandroid:generateDebugResValues UP-TO-DATE
:rxfirebaseandroid:generateDebugResources UP-TO-DATE
:rxfirebaseandroid:packageDebugResources UP-TO-DATE
:rxfirebaseandroid:processDebugManifest UP-TO-DATE
:rxfirebaseandroid:processDebugResources UP-TO-DATE
:rxfirebaseandroid:generateDebugSources UP-TO-DATE
:rxfirebaseandroid:processDebugJavaRes UP-TO-DATE
:rxfirebaseandroid:compileDebugJavaWithJavac UP-TO-DATE
:rxfirebaseandroid:extractDebugAnnotations UP-TO-DATE
:rxfirebaseandroid:mergeDebugProguardFiles UP-TO-DATE
:rxfirebaseandroid:packageDebugJar UP-TO-DATE
:rxfirebaseandroid:compileDebugNdk UP-TO-DATE
:rxfirebaseandroid:packageDebugJniLibs UP-TO-DATE
:rxfirebaseandroid:packageDebugLocalJar UP-TO-DATE
:rxfirebaseandroid:packageDebugRenderscript UP-TO-DATE
:rxfirebaseandroid:bundleDebug UP-TO-DATE
:rxfirebaseandroid:compileDebugSources UP-TO-DATE
:rxfirebaseandroid:assembleDebug UP-TO-DATE
BUILD SUCCESSFUL
Total time: 10.146 secs
Archiving cache...
INFO[0000] Creating archive cache.tgz ...
INFO[0002] Done!
Archiving artifacts...
INFO[0000] Creating archive artifacts.tgz ...
INFO[0000] Done!
Uploading artifacts...
20.0K artifacts.tgz
Build succeeded.
https://stackoverflow.com/a/35478988/2026105 https://stackoverflow.com/a/35478988/2026105
Turns out another post has the answer to this! 原来另一篇文章有这个答案! There is a work around which is to run a
before_script
that sets the GRADLE_HOME
to the directory of your project, which you can then slurp up with the cache directive. 有一个工作是运行一个
before_script
, GRADLE_HOME
设置为项目目录,然后您可以使用cache指令进行GRADLE_HOME
。
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
cache:
paths:
- .gradle/wrapper
- .gradle/caches
build:
stage: build
script:
- ./gradlew assemble
test:
stage: test
script:
- ./gradlew check
According to Kamil Trzciński 根据KamilTrzciński的说法
We run the caching outside the build container.
我们在构建容器外部运行缓存。 So you can cache absolute paths, but since only /builds and /cache is transferred you have no-way to cache container-local files.
因此,您可以缓存绝对路径,但由于只传输了/ builds和/ cache,因此您无法缓存容器本地文件。
That means you need to put the gradle files into eg the /cache
folder 这意味着您需要将gradle文件放入例如
/cache
文件夹中
before_script:
- export GRADLE_USER_HOME=cache/.gradle
...
cache:
key: "$CI_COMMIT_REF_NAME" #optional: per branch caching, can be omitted
paths:
- $GRADLE_USER_HOME/caches/
- $GRADLE_USER_HOME/wrapper/
- $GRADLE_USER_HOME/build-cache/
The /cache
folder will be kept automatically, but I keep the cache config for clarity. /cache
文件夹将自动保留,但为了清楚起见,我保留了缓存配置。
We're running inside docker container on shared runner infrastructure using minio S3 backend to distribute the cache and landed on a solution somewhere along these lines:我们在共享运行器基础设施上的 docker 容器内运行,使用 minio S3 后端分发缓存,并在以下几处找到解决方案:
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false -Dorg.gradle.vfs.watch=false -Dorg.gradle.caching=true"
GRADLE_USER_HOME: .gradle/
default:
image: yourfavoritegradlecontainer:latest
cache: &gradle-cache
key:
files:
- gradle/wrapper/gradle-wrapper.properties
paths:
- .gradle/caches/
- .gradle/notifications/
- .gradle/wrapper/
policy: pull
build:
stage: build
cache:
<<: *gradle-cache
policy: pull-push
script:
- ./gradlew assemble
artifacts:
paths:
- app/build/outputs/apk
test:
stage: test
script:
- ./gradlew test
The cache gets invalidated, when the gradle-wrapper.properties
file is changed and to prevent cache thrashing, only build job is allowed to upload the cache.缓存失效,当
gradle-wrapper.properties
文件更改并防止缓存抖动时,只允许构建作业上传缓存。 Of course, you could always add key: "$CI_COMMIT_REF_NAME"
to invalidate the cache more aggressively.当然,您总是可以添加
key: "$CI_COMMIT_REF_NAME"
以更积极地使缓存无效。
Seperately we export build artifacts, it supposedly helps, too.我们单独导出构建工件,据说它也有帮助。
GitLab is taking care of storing and restoring the cache, automatically from inside the container. GitLab 负责从容器内部自动存储和恢复缓存。 This even works for a single shared runner.
这甚至适用于单个共享跑步者。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.