无法使用雅典（Go 模块代理）对私有模块进行身份验证

Question

I was testing Athens with docker in my local, trying to setup the authentication so that Athens is able to access to company's private module. I followed the guide here and the command go get github.com/SomeCompany/someprivatemodule successfully pulled the latest version of that lib to my local. But there's something weird.

1. Those config files (.gitconfig, .ssh/config and.ssh/id_rsa) didn't make that command work. Instead, the .netrc file, consisting of my GitHub token, makes it work. I've tried to run the docker image without mounting those config files and the command still works; However, if I removed ~/.netrc in my local, the go get command fails. It seems go get is still depending on local config instead of Athens.

go get command fails when .netrc removed

2. The go get command does pull the private module from GitHub. However, under ATHENS_DISK_STORAGE_ROOT , I can only find those public modules that github.com/SomeCompany/someprivatemodule depends on, and those private modules can only be found under GOPATH /pkg/mod. Isn't Athens supposed to download the private module to ATHENS_DISK_STORAGE_ROOT , so that once GitHub/gitlab is down, users can still download those private modules from Athens?

Missing private module in Athens dir

Here is my setup:

GOPROXY=127.0.0.1:3000, which is the exposed port of Athens. GOPRIVATE="github.com/SomeCompany"
GONOPROXY="github.com/SomeCompany"
GONOSUMDB="github.com/SomeCompany"

~/.netrc

machine github.com login some_personal_access_token

.gitconfig

[url "ssh://git@github.com/"]
    insteadOf = https://github.com/
[url "ssh://git@gitlab.com/"]
    insteadOf = https://gitlab.com/

.ssh/config

Host github.com
Hostname github.com
StrictHostKeyChecking no
IdentityFile /root/.ssh/id_rsa

command of running docker image

docker run -d -v $ATHENS_STORAGE:/var/lib/athens -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk -v $PWD/.gitconfig:/root/.gitconfig -v @PWD/.ssh:/root/.ssh --name athens-proxy --restart always -p 3000:3000 gomods/athens:v0.11.0

1st edit

By setting GOPRIVATE to empty, GONOPROXY and GONOSUMDB get unset automatically. And I tried to rerun the docker with original setup, I got the 404 error:

404 error in my terminal

Docker log:

INFO[2:35PM]: Exporter not specified. Traces won't be exported

2021-05-29 14:35:11.957339 I | Starting application at port :3000

INFO[2:35PM]: exit status 1: go list -m: github.com/SomeCompany@latest: invalid github.com/ import path "github.com/SomeCompany"

http-method=GET http-path=/github.com/SomeCompany/@v/list kind=Not Found module= operation=download.ListHandler ops=[download.ListHandler pool.List protocol.List vcsLister.List] request-id=3660faa3-256f-43fb-9937-f29565e7afa9 version=

INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/SomeCompany/@v/list http-status=404 request-id=3660faa3-256f-43fb-9937-f29565e7afa9

INFO[2:35PM]: exit status 1: go list -m: github.com@latest: unrecognized import path "github.com": parse https://github.com/?go-get=1: no go-import meta tags ()

http-method=GET http-path=/github.com/@v/list kind=Not Found module= operation=download.ListHandler ops=[download.ListHandler pool.List protocol.List vcsLister.List] request-id=4d6659d2-710d-445f-b07f-7407aa4f2e3e version=

INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/@v/list http-status=404 request-id=4d6659d2-710d-445f-b07f-7407aa4f2e3e

INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/list http-status=200 request-id=e8d95d77-81dc-46fe-9523-e35f6dc8201f

DEBUG[2:35PM]: saving github.com/SomeCompany/someprivatemodule@v1.0.5 to storage... http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info request-id=24964b9c-b0b2-450a-bd9a-9d59f60a5c52

INFO[2:35PM]: github.com/SomeCompany/someprivatemodule@v1.0.5/go.mod: verifying module: github.com/SomeCompany/someprivatemodule@v1.0.5/go.mod: reading https://sum.golang.org/lookup/github.com/SomeCompany/someprivatemodule@v1.0.5: 410 Gone

server response: not found: github.com/SomeCompany/someprivatermodule@v1.0.5: invalid version: unknown revision v1.0.5 http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info kind=Not Found module=github.com/SomeCompany/someprivatemodule operation=download.InfoHandler ops=[download.InfoHandler pool.Info protocol.Info protocol.processDownload stash.Pool stasher.Stash stasher.fetchModule goGetFetcher.Fetch module.downloadModule] request-id=24964b9c-b0b2-450a-bd9a-9d59f60a5c52 version=v1.0.5

INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info http-status=404 request-id=24964b9c-b0b2-450a-bd9a-9d59f60a5c52

It seems Athens tries to find packages from sum.golang.org. So I tried to mount .netrc file with proper credentials and set ATHENS_NETRC_PATH. This still gives me same error.

Another fact: with v0.11.0, I couldn't download anything with go get. However, with v0.2.0 version of Athens docker image, I was able to download part of the modules. The docker log (v0.2.0) is shown below:

buffalo: Unless you set SESSION_SECRET env variable, your session storage is not protected!

time="2021-05-29T19:39:20Z" level=info msg="Exporter not specified. Traces won't be exported"

buffalo: Starting application at :3000

time="2021-05-29T19:39:26Z" level=error msg="exit status 1: go list -m github.com/SomeCompany: invalid github.com/ import path \"github.com/SomeCompany\"\n" http-method=GET http-path="/github.com/SomeCompany/@v/list/" http-url="/github.com/SomeCompany/@v/list/" kind="Internal Server Error" module= operation=download.ListHandler ops="[download.ListHandler pool.List protocol.List vcsLister.List]" version=

handler: GET /github.com/SomeCompany/@v/list/ [500]

time="2021-05-29T19:39:26Z" level=error msg="exit status 1: go list -m github.com: unrecognized import path \"github.com\" (parse https://github.com?go-get=1: no go-import meta tags ())\n" http-method=GET http-path=/github.com/@v/list/ http-url=/github.com/@v/list/ kind="Internal Server Error" module= operation=download.ListHandler ops="[download.ListHandler pool.List protocol.List vcsLister.List]" version=

handler: GET /github.com/@v/list/ [500]

handler: GET /github.com/SomeCompany/someprivatemodule/@v/list/ [200]

time="2021-05-29T19:39:26Z" level=error msg="exit status 1: go: finding github.com/SomeCompany/someprivatemodule/somefile latest\ngo list -m github.com/SomeCompany/someprivatemodule/somefile: no matching versions for query \"latest\"\n" http-method=GET http-path="/github.com/SomeCompany/someprivatemodule/somefile/@v/list/" http-url="/github.com/SomeCompany/someprivatemodule/somefile/@v/list/" kind="Internal Server Error" module= operation=download.ListHandler ops="[download.ListHandler pool.List protocol.List vcsLister.List]" version=

handler: GET /github.com/SomeCompany/someprivatemodule/ratelimit/@v/list/ [500]

handler: GET /github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info [200]

handler: GET /github.com/SomeCompany/someprivatemodule/@v/v1.0.5.mod [200]

handler: GET /github.com/SomeCompany/someprivatemodule/@v/v1.0.5.zip [200]

And apparently this still won't work cuz its missing part of the module.

Edit: Solved by adding GONOSUM_PATTERNS

---

Answer 1

I just tried Athens a bit. It serves as a Go module cache. When you start athens and set GOPROXY=127.0.0.1:3000, the go get command would try to get modules using the sequence below:

1. check GOPATH/pkg/mod
2. request from GOPROXY

Here your GOPROXY is athens server, thus your local go get will send request to athens. Then athens will check if the module is in ATHENS_DISK_STORAGE_ROOT; if not, get the proxy from github.com or wherever it should be.

You can try to walkthrough the process.

As for your questions,

I notice that you set github.com/SomeCompany in your GONOPROXY, thus all modules in this repository will not use GOPROXY, and athens will not work as well.

That's why in Question1, your .netrc is used; and in Question2, only the modules that github.com/SomeCompany/someprivatemodule depends on is downloaded from athens, but the module itself is not.

Remove the GONOPROXY and try again, see what's happening.