Hyperledger Fabric: TLS Internal Error using Fabric Gateway Java SDK

Question

Fabric Version: v2.2.4 Fabric Gateway Version: v2.2.0

Currently, I am not using Fabric CA and docker. Instead, I am using OpenSSL to generate my own certificates. There is no problem using command line to join channel or invoke chaincode with TLS enabled (Server TLS).

However, when I try to invoke my chaincode using Fabric Gateway Java SDK with TLS, the peer shows:

2021-10-20 23:53:51.571 EDT [core.comm] ServerHandshake -> ERRO 411 Server TLS handshake failed in 731.664253ms with error remote error: tls: internal error server=PeerServer remoteaddress=127.0.0.1:48920
2021-10-20 23:53:51.648 EDT [core.comm] ServerHandshake -> ERRO 412 Server TLS handshake failed in 17.623962ms with error remote error: tls: internal error server=PeerServer remoteaddress=127.0.0.1:48924
2021-10-20 23:53:52.389 EDT [core.comm] ServerHandshake -> ERRO 413 Server TLS handshake failed in 31.834126ms with error remote error: tls: internal error server=PeerServer remoteaddress=127.0.0.1:48928
2021-10-20 23:53:53.013 EDT [core.comm] ServerHandshake -> ERRO 414 Server TLS handshake failed in 20.97883ms with error remote error: tls: internal error server=PeerServer remoteaddress=127.0.0.1:48932
2021-10-20 23:53:53.453 EDT [core.comm] ServerHandshake -> ERRO 415 Server TLS handshake failed in 27.840631ms with error remote error: tls: internal error server=PeerServer remoteaddress=127.0.0.1:48936

if I disable the TLS, the chaincode can be invoked and queried but it shows an error message and display the certificate of the peer:

04:45:56.020 [main] ERROR org.hyperledger.fabric.sdk.security.CryptoPrimitives - Cannot 
validate certificate. Error is: Path does not chain with any of the trust anchors
Certificate[
[
  Version: V3
  Subject: CN=peer-telecom, OU=peer, O=peer0.telecom.com, ST=Wilayah Persekutuan Kuala Lumpur, C=MY
  Signature Algorithm: SHA256withECDSA, OID = 1.2.840.10045.4.3.2

  Key:  Sun EC public key, 256 bits
  public x coord: 63258922835963897769642318382353579773301130246303245867091942631050654760639
  public y coord: 17014436126955018341557373411142402131278326611630973049174140241981148702177
  parameters: secp256r1 [NIST P-256, X9.62 prime256v1] (1.2.840.10045.3.1.7)
  Validity: [From: Mon Oct 18 04:30:10 EDT 2021,
               To: Tue Oct 18 04:30:10 EDT 2022]
  Issuer: CN=hyperledger-telecom, O=ica.hyperledger.telecom.com, ST=Wilayah Persekutuan Kuala Lumpur, C=MY
  SerialNumber: [    1000]

Certificate Extensions: 4
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 70 2B 57 B5 7C 6A 38 DE   AB 6F DD 7E C4 63 FE 39  p+W..j8..o...c.9
0010: 54 22 D9 F9                                        T"..
]
]

[2]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[3]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
]

[4]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 62 C2 4A FB 13 54 44 DF   15 AB 2B 09 78 4F 79 4A  b.J..TD...+.xOyJ
0010: CB 37 1D D7                                        .7..
]
]

]
  Algorithm: [SHA256withECDSA]
  Signature:
0000: 30 44 02 20 2D 20 15 9D   53 D4 DE EF 56 0D E6 6D  0D. - ..S...V..m
0010: 04 DA 99 F8 0C AC D5 A7   87 66 51 04 23 A0 4D C2  .........fQ.#.M.
0020: C8 98 95 5C 02 20 5C A5   5A 2D 19 43 FA E8 C0 E1  ...\. \.Z-.C....
0030: 49 4E C0 DF C9 59 F8 10   34 D6 94 05 51 38 E9 17  IN...Y..4...Q8..
0040: C5 F1 20 1F 0C EC                                  .. ...

]

Java Application code:

package org.example.contract;

import java.io.IOException;
import java.io.Reader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.Files;

import java.util.concurrent.TimeoutException;
import java.security.InvalidKeyException;
import java.security.PrivateKey;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;


import org.hyperledger.fabric.gateway.Identities;
import org.hyperledger.fabric.gateway.Identity;
import org.hyperledger.fabric.gateway.Contract;
import org.hyperledger.fabric.gateway.ContractException;
import org.hyperledger.fabric.gateway.Gateway;
import org.hyperledger.fabric.gateway.Network;
import org.hyperledger.fabric.gateway.Wallet;
import org.hyperledger.fabric.gateway.Wallets;

public class Sample {
    
    X509Certificate[] clientCert = null;
    PrivateKey clientKey = null;

    private static X509Certificate readX509Certificate(final Path certificatePath) throws IOException, CertificateException {
        try (Reader certificateReader = Files.newBufferedReader(certificatePath, StandardCharsets.UTF_8)) {
            return Identities.readX509Certificate(certificateReader);
        }
    }

    private static PrivateKey getPrivateKey(final Path privateKeyPath) throws IOException, InvalidKeyException {
        try (Reader privateKeyReader = Files.newBufferedReader(privateKeyPath, StandardCharsets.UTF_8)) {
            return Identities.readPrivateKey(privateKeyReader);
        }
    }

    public static void main(String[] args) throws IOException {
        // Load an existing wallet holding identities used to access the network.
        // A wallet stores a collection of identities
        Path walletPath = Paths.get(".", "wallet");
        Wallet wallet = Wallets.newFileSystemWallet(walletPath);
        try {
                  
            Path credentialPath = Paths.get("/root","fabric","localtls","crypto-config", "peerOrganizations",
                "org1.telecom.com", "users", "Admin@org1.telecom.com", "msp");
            System.out.println("credentialPath: " + credentialPath.toString());
            Path certificatePath = credentialPath.resolve(Paths.get("signcerts",
                "Admin@org1.telecom.com-cert.pem"));
            System.out.println("certificatePem: " + certificatePath.toString());
            Path privateKeyPath = credentialPath.resolve(Paths.get("keystore",
                "Admin@telecom.com.key"));
      
            X509Certificate certificate = readX509Certificate(certificatePath);
            PrivateKey privateKey = getPrivateKey(privateKeyPath);
      
            Identity identity = Identities.newX509Identity("Org1MSP", certificate, privateKey);
            
            String identityLabel = "Admin@org1.telecom.com";
            wallet.put(identityLabel, identity);
      
            System.out.println("Write wallet info into " + walletPath.toString() + " successfully.");
      
        } catch (IOException | CertificateException | InvalidKeyException e) {
            System.err.println("Error adding to wallet");
            e.printStackTrace();
        }
        // Path to a common connection profile describing the network.
        Path networkConfigFile = Paths.get("/root","fabric","localtls","connection.yaml");

        String userName = "Admin@org1.telecom.com";
        // Configure the gateway connection used to access the network.,
        Gateway.Builder builder = Gateway.createBuilder()
                .identity(wallet, userName)
                .networkConfig(networkConfigFile);

        // Create a gateway connection
        try (Gateway gateway = builder.connect()) {

            // Obtain a smart contract deployed on the network.
            Network network = gateway.getNetwork("mychannel");
            Contract contract = network.getContract("chaincode");

            // Submit transactions that store state to the ledger.
            byte[] createCarResult = contract.createTransaction("createMyAsset")
                    .submit("CAR", "Honda");
            System.out.println(new String(createCarResult, StandardCharsets.UTF_8));

            byte[] queryCar = contract.submitTransaction("readMyAsset", "CAR");
            System.out.println(new String(queryCar, StandardCharsets.UTF_8));

        } catch (ContractException | TimeoutException | InterruptedException e) {
            e.printStackTrace();
        }
    }
}

connection.yaml:

name: "Network-Config-Test"
description: "The network used in the integration tests"
version: 1.0.0
client:
  organization: telecom

organizations:
  telecom:
    mspid: Org1MSP

    peers:
      - peer0.org1.telecom.com

    #adminPrivateKey:
      #path: '/root/fabric/localtls/crypto-config/peerOrganizations/org1.telecom.com/users/Admin@org1.telecom.com/msp/keystore/Admin@telecom.com.key'

    #signedCert:
      #path: '/root/fabric/localtls/crypto-config/peerOrganizations/org1.telecom.com/users/Admin@org1.telecom.com/msp/signcerts/Admin@org1.telecom.com-cert.pem'
      
orderers:
  orderer.example.com:
    url: grpcs://localhost:6050

   
      #client: 
        #keyfile: '/root/fabric/localtls/crypto-config/peerOrganizations/org1.telecom.com/users/Admin@org1.telecom.com/tls/client-o.key'
        #certfile: '/root/fabric/localtls/crypto-config/peerOrganizations/org1.telecom.com/users/Admin@org1.telecom.com/tls/client-o.crt'

    grpcOptions:
      hostnameOverride: orderer.example.com
      grpc-max-send-message-length: 15
      grpc.keepalive_time_ms: 360000
      grpc.keepalive_timeout_ms: 180000
      negotiationType: TLS
      sslProvider: openSSL

    tlsCACerts:
      path: /root/fabric/localtls/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/ca.crt


peers:
  peer0.org1.telecom.com:
    url: grpcs://localhost:7051
      #client: 
        #keyfile: '/root/fabric/localtls/crypto-config/peerOrganizations/org1.telecom.com/users/Admin@org1.telecom.com/tls/client.key'
        #certfile: '/root/fabric/localtls/crypto-config/peerOrganizations/org1.telecom.com/users/Admin@org1.telecom.com/tls/client.crt'

    grpcOptions:
      ssl-target-name-override: peer0.org1.telecom.com
      grpc.http2.keepalive_time: 15
      hostnameOverride: peer0.org1.telecom.com
      negotiationType: TLS
      sslProvider: openSSL
    
    tlsCACerts:
      path: /root/fabric/localtls/crypto-config/peerOrganizations/org1.telecom.com/peers/peer0.org1.telecom.com/tls/ca.crt

What am I missing? Any answer is welcomed!

Answer 1

Fabric TLS certificates are very fiddly . For instance, we've found several cases where the peer and chaincode will accept a malformed cert, but the gateway client refuses to connect using the same certificate. What is probably happening is that you have a problem with the openssl generated cert, but the peer is too lax to report the error. (It also doesn't help that the client TLS libraries simply disconnect, without reporting on the root failure during the handshake.)

In the example above, the cert output shows some differences in PathLen and KeyUsage attributes that differ from what are normally generated by a Fabric CA.

Here are some ideas and techniques that we've found to be useful in debugging TLS handshake issues with the gateway / client SDKs. Try to use these techniques to close any gaps between your certs and the reference certs generated by the Fabric CAs:

• Use the Fabric CAs. Even if you plan to generate a certificate chain using OpenSSL and an external authority, you can use the Fabric CAs to generate TLS enrollments and certificates, and compare the reference certs against what you have built up with OpenSSL.

• Use curl , or other TLS-enabled clients to help verify the correctness of a certificate. In many cases the errors output by an independent client are directly applicable to the failed TLS handshake when connecting the fabric client.

• Use the test-network-k8s system as a reference for setting up the TLS and CA infrastructure. In addition to a "push button" test network, this will provide CA endpoints that can be used to generate reference enrollments / certificates for study.

• Inspect the certificates generated by the Fabric CAs, and compare against your hand-crafted certs. For example, here is a certificate dump from a TLS cert generated with the Kube test network - make sure your OpenSSL certs have the same, or similar feature sets and attributes.

$ openssl x509 -in /tmp/ca-cert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            4c:63:74:d5:99:29:ce:e0:b6:28:a2:b5:a4:0e:a0:c1:f3:e9:a9:d5
    Signature Algorithm: ecdsa-with-SHA256
        Issuer: C=US, ST=North Carolina, O=Hyperledger, OU=Fabric, CN=fabric-ca-server
        Validity
            Not Before: Oct 21 10:36:00 2021 GMT
            Not After : Oct 17 10:36:00 2036 GMT
        Subject: C=US, ST=North Carolina, O=Hyperledger, OU=Fabric, CN=fabric-ca-server
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub: 
                    04:53:f4:ad:e6:6b:4c:75:7e:4a:6d:6e:cd:73:b0:
                    81:a8:d7:d7:55:c0:fd:22:92:15:fc:2d:20:44:c6:
                    ec:55:c9:cc:88:3a:14:09:77:e5:4f:4b:b8:98:ee:
                    71:09:da:e6:f8:7c:f7:39:fa:41:fc:f3:a2:fe:a4:
                    1e:34:ec:a9:b5
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Subject Key Identifier: 
                AB:AF:85:A3:D3:2E:9A:A9:03:49:F5:5C:30:32:2B:92:EC:92:B3:D0
            X509v3 Subject Alternative Name: 
                IP Address:127.0.0.1
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:bf:cc:c1:d2:29:b1:04:3f:55:31:c6:b7:69:
         ca:72:12:d7:67:55:14:cd:23:f7:75:16:6c:b1:63:7f:e6:9c:
         24:02:20:5d:ff:e3:7e:84:22:d3:f3:52:bd:96:fa:dc:2d:94:
         2f:6b:a3:bc:ab:3e:b3:87:10:fd:30:51:a2:4a:ca:ce:b4
-----BEGIN CERTIFICATE-----
MIICKDCCAc6gAwIBAgIUTGN01ZkpzuC2KKK1pA6gwfPpqdUwCgYIKoZIzj0EAwIw
aDELMAkGA1UEBhMCVVMxFzAVBgNVBAgTDk5vcnRoIENhcm9saW5hMRQwEgYDVQQK
EwtIeXBlcmxlZGdlcjEPMA0GA1UECxMGRmFicmljMRkwFwYDVQQDExBmYWJyaWMt
Y2Etc2VydmVyMB4XDTIxMTAyMTEwMzYwMFoXDTM2MTAxNzEwMzYwMFowaDELMAkG
A1UEBhMCVVMxFzAVBgNVBAgTDk5vcnRoIENhcm9saW5hMRQwEgYDVQQKEwtIeXBl
cmxlZGdlcjEPMA0GA1UECxMGRmFicmljMRkwFwYDVQQDExBmYWJyaWMtY2Etc2Vy
dmVyMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEU/St5mtMdX5KbW7Nc7CBqNfX
VcD9IpIV/C0gRMbsVcnMiDoUCXflT0u4mO5xCdrm+Hz3OfpB/POi/qQeNOyptaNW
MFQwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQEwHQYDVR0OBBYE
FKuvhaPTLpqpA0n1XDAyK5LskrPQMA8GA1UdEQQIMAaHBH8AAAEwCgYIKoZIzj0E
AwIDSAAwRQIhAL/MwdIpsQQ/VTHGt2nKchLXZ1UUzSP3dRZssWN/5pwkAiBd/+N+
hCLT81K9lvrcLZQva6O8qz6zhxD9MFGiSsrOtA==
-----END CERTIFICATE-----