X509: certfificate signed by unknown authority when using docker login from a remote machine

Question

I'm trying to acces a private nexus repo. using docker login from a remote machine on the same network and despite i have followed instuctions in the documentation of docker i still get the x509: certfificate signed by unknown authority error, I'm on a centOs 8 machine, with nexus OSS 3.29.2-02 , i've configured the the repo according to the following documentation Configuring SSL and i've created the self signed certificate using java keytool and it works when i access it from the browser on the same machine and from the remote machine, in the /app/sonatype-work/nexus3/log/nexus.log log file there is no errors. i've copied the certificate.cer to the /etc/docker/certs.d/domain:port/ location then i also copied it to /etc/pki/ca-trust/source/anchors/ and ran sudo update-ca-trust according to docker docs: docker insecure registry when i keytool -printcert -sslserver domain:port -v from the remote machine the certificate is printed. when i wget from the remote machine it works and the certificate is successfully validated and data downloaded. i've checked many topics but all are talking about putting the certificate in the above mentioned locations what i've already done. thanks in advance.

----------------------update--------------------------

[mehdilapin@localhost ~]$ wget https://mycustomregistry.com:7575
--2021-04-11 14:53:59--  https://mycustomregistry.com:7575/
Auflösen des Hostnamens mycustomregistry.com (mycustomregistry.com)… 192.168.1.9
Verbindungsaufbau zu mycustomregistry.com (mycustomregistry.com)|192.168.1.9|:7575 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 8903 (8,7K) [text/html]
Wird in »index.html.2« gespeichert.

index.html.2                                         100%[=====================================================================================================================>]   8,69K  --.-KB/s    in 0s      

2021-04-11 14:53:59 (287 MB/s) - »index.html.2« gespeichert [8903/8903]

[mehdilapin@localhost ~]$ sudo keytool -printcert -sslserver mycustomregistry.com:4563/registry/api -v
Certificate #0
====================================
Eigentümer: CN=mycustomregistry.com, OU=organization Unit, O=organization, L=USA, ST=NewYork, C=US
Aussteller: CN=mycustomregistry.com, OU=organization Unit, O=organization, L=USA, ST=NewYork, C=US
Seriennummer: 68e917a2
Gültig von: Sun Apr 11 14:34:54 CET 2021 bis: Mon Apr 11 14:34:54 CET 2022
Zertifikatsfingerprints:
     SHA1: 05:95:71:99:93:D1:30:A0:D1:82:0C:73:61:47:69:F0:2A:A4:52:B3
     SHA256: EA:8A:0A:0C:C6:4B:BE:73:57:78:CC:DC:08:DE:92:8E:04:6F:B8:3E:8F:2A:71:C8:AD:5A:E7:19:BB:31:7C:AE
Signaturalgorithmusname: SHA256withRSA
Public Key-Algorithmus von Subject: 2048-Bit-RSA-Schlüssel
Version: 3

Erweiterungen: 

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: EF C5 91 38 62 ED 54 12   4E AA 9C 0F C6 73 F2 0C  ...8b.T.N....s..
0010: 83 BF CA 5E                                        ...^
]
]

[mehdilapin@localhost ~]$ sudo keytool -printcert -file /etc/docker/certs.d/mycustomregistry.com\:4563/ca-certificate.cer 
Eigentümer: CN=mycustomregistry.com, OU=organization Unit, O=organization, L=USA, ST=NewYork, C=US
Aussteller: CN=mycustomregistry.com, OU=organization Unit, O=organization, L=USA, ST=NewYork, C=US
Seriennummer: 68e917a2
Gültig von: Sun Apr 11 14:34:54 CET 2021 bis: Mon Apr 11 14:34:54 CET 2022
Zertifikatsfingerprints:
     SHA1: 05:95:71:99:93:D1:30:A0:D1:82:0C:73:61:47:69:F0:2A:A4:52:B3
     SHA256: EA:8A:0A:0C:C6:4B:BE:73:57:78:CC:DC:08:DE:92:8E:04:6F:B8:3E:8F:2A:71:C8:AD:5A:E7:19:BB:31:7C:AE
Signaturalgorithmusname: SHA256withRSA
Public Key-Algorithmus von Subject: 2048-Bit-RSA-Schlüssel
Version: 3

Erweiterungen: 

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: EF C5 91 38 62 ED 54 12   4E AA 9C 0F C6 73 F2 0C  ...8b.T.N....s..
0010: 83 BF CA 5E                                        ...^
]
]


[mehdilapin@localhost ~]$ sudo keytool -printcert -file /etc/pki/ca-trust/source/anchors/ca-certificate.cer -v
Eigentümer: CN=mycustomregistry.com, OU=organization Unit, O=organization, L=USA, ST=NewYork, C=US
Aussteller: CN=mycustomregistry.com, OU=organization Unit, O=organization, L=USA, ST=NewYork, C=US
Seriennummer: 68e917a2
Gültig von: Sun Apr 11 14:34:54 CET 2021 bis: Mon Apr 11 14:34:54 CET 2022
Zertifikatsfingerprints:
     SHA1: 05:95:71:99:93:D1:30:A0:D1:82:0C:73:61:47:69:F0:2A:A4:52:B3
     SHA256: EA:8A:0A:0C:C6:4B:BE:73:57:78:CC:DC:08:DE:92:8E:04:6F:B8:3E:8F:2A:71:C8:AD:5A:E7:19:BB:31:7C:AE
Signaturalgorithmusname: SHA256withRSA
Public Key-Algorithmus von Subject: 2048-Bit-RSA-Schlüssel
Version: 3

Erweiterungen: 

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: EF C5 91 38 62 ED 54 12   4E AA 9C 0F C6 73 F2 0C  ...8b.T.N....s..
0010: 83 BF CA 5E                                        ...^
]
]

Answer 1

Certificates in /etc/docker/certs.d/ need to be x509 formatted and named with a crt extension (it's actually possible to configure client tls settings with this same folder). So rename:

/etc/docker/certs.d/mycustomregistry.com\:4563/ca-certificate.cer

to an x509/pem formatted certificate named:

/etc/docker/certs.d/mycustomregistry.com\:4563/ca-certificate.crt

That doesn't explain why the OS certificates aren't working. When checking with wget and curl, you should be able to reach the v2 api, even if it gives you a permission denied error:

curl https://mycustomregistry.com:4563/v2/