Composer - cannot download flex recipes JSON file - 404 vs. curl 200 OK

Question

Today I wanted to launch my old Symfony flex based project (I have removed the docker containers/images/cache locally some time ago). When launching the container, in the entry point, composer is supposed to install all dependencies, but it fails. I've launched the project without installing dependencies, so I could run it manually, and a strange thing happens. Composer is unable to download the recipes index.json file, while I can do it with curl.

Here are composer logs:

# composer update symfony/flex

In CurlDownloader.php line 623:

  The "https://raw.githubusercontent.com/symfony/recipes-contrib/flex/main/index.json" file could not be downloaded (HTTP/2 404 )


update [--with WITH] [--prefer-source] [--prefer-dist] [--prefer-install PREFER-INSTALL] [--dry-run] [--dev] [--no-dev] [--lock] [--no-install] [--no-audit] [--audit-format AUDIT-FORMAT] [--no-autoloader] [--no-suggest] [--no-progress] [-w|--with-dependencies] [-W|--with-all-dependencies] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--apcu-autoloader-prefix APCU-AUTOLOADER-PREFIX] [--ignore-platform-req IGNORE-PLATFORM-REQ] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [-i|--interactive] [--root-reqs] [--] [<packages>...]

And here's curl:

# curl -IL --http2 https://raw.githubusercontent.com/symfony/recipes/flex/main/index.json
HTTP/2 200
cache-control: max-age=300
content-security-policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
content-type: text/plain; charset=utf-8
etag: "cbf7d5d470e448beeb667739f56cc627eb271aa54de7d1a46d10eb8ee72ddc36"
strict-transport-security: max-age=31536000
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
x-github-request-id: D29A:D902:9C967E:ACDA5B:63B2C9F7
accept-ranges: bytes
date: Mon, 02 Jan 2023 13:47:19 GMT
via: 1.1 varnish
x-served-by: cache-fra-eddf8230085-FRA
x-cache: HIT
x-cache-hits: 1
x-timer: S1672667240.634749,VS0,VE1
vary: Authorization,Accept-Encoding,Origin
access-control-allow-origin: *
x-fastly-request-id: aac3282f232b3395c82294f6b70b799be6586a24
expires: Mon, 02 Jan 2023 13:52:19 GMT
source-age: 74
content-length: 73467

I ran the composer diagnose command:

# composer diagnose
Checking composer.json: WARNING
require.symfony/one-signal-notifier : exact version constraints (6.1.x-dev) should be avoided if the package follows semantic versioning
Checking platform settings: OK
Checking git settings: OK git version 2.38.2
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com oauth access: The oauth token for github.com seems invalid, run "composer config --global --unset github-oauth.github.com" to remove it
Checking disk free space: OK
Checking pubkeys: FAIL
Missing pubkey for tags verification
Missing pubkey for dev verification
Run composer self-update --update-keys to set them up
Checking composer version: OK
Composer version: 2.5.1
PHP version: 8.1.13
PHP binary path: /usr/local/bin/php
OpenSSL version: OpenSSL 3.0.7 1 Nov 2022
cURL version: 7.86.0 libz 1.2.13 ssl OpenSSL/3.0.7
zip: extension present, unzip present, 7-Zip not available

Answer 1

TL;DR Check if you have a custom authentication configured for github/gitlab or other services in your project. It may be outdated and could cause the above issue. In my case, it was configured in auth.json file.

Story mode:

I've noticed the The oauth token for github.com seems invalid, run "composer config --global --unset github-oauth.github.com" to remove it . It was the only clue I had.

I've tried running the composer config --global --unset github-oauth.github.com command but it changed nothing, so I've ignored it.

After a while, trying to figure it out (and running composer install on a server) I noticed an auth.json file added to the repository. It seems that the file contained an outdated GitHub OAuth key required by a custom package we used to use some time ago. Since then our PR was merged into the official package and our version was no longer used, but we didn't remove the old auth.json file.

EDIT: Now in Github, you have the ability to generate fine-grained tokens with an expiration date of up to 1 year.