Hyperledger Fabric 2.3 连接问题

Question

Facing an issue while querying ledger. Here is the how our network is laid out. There are 2 orgs on a kubernetes cluster inside the corp network and one on a docker swarm on an azure vm inside the network too. The azure vm node and the k8s cluster nodes communicate with each other over an nginx server. Now the reason behind this elaborate setup is because our supplychain usecase requires partners from different companies to join our network. So to simulate an external partner outside the corp network we use the azure vm. Since we plan to productionize the implementation we couldn't use Fabric crypto config generated certificates and got new certificates issued using our company's intermediate and root certs. Now there are chaincodes installed on this network setup with endorsement policy enabled that works perfectly on all 3 nodes. We are using Fabric 2.3.0

Now the first issue that I had faced, was the TLS certificate to use in the connection.json file. That was solved by chaining the certificates as described in the SO post here . The current issue is that the nodejs code is able to connect to the orgs but is unable to execute any read or write operations. In the JS code below if I uncomment the console logs of the channel.getPeer() responses it prints the entire peer object properly.

Here is my connection.json. All the 10.100.xx.xx ips are of the pods in the k8s cluster and the public.ip.address is that of the nginx server

{
    "name": "byfn",
    "version": "1.0.0",
    "client": {
        "organization": "ORG2MSP",
        "connection": {
            "timeout": {
                "peer": {
                    "endorser": "10000"
                },
                "orderer": "10000"
            }
        }
    },
    "channels": {
        "supplychain": {
            "orderers": [
                "ord1.orderers.org1.com",
                "ord2.orderers.org1.com",
                "ord3.orderers.org1.com"
            ],
            "peers": {
                "peer1.peers.org1.com": {
                    "endorsingPeer": true,
                    "chaincodeQuery": true,
                    "ledgerQuery": true,
                    "eventSource": true
                },
                "peer1.peers.org3.com": {
                    "endorsingPeer": true,
                    "chaincodeQuery": true,
                    "ledgerQuery": true,
                    "eventSource": true
                },
                "peer1.peers.org2.com": {
                    "endorsingPeer": true,
                    "chaincodeQuery": true,
                    "ledgerQuery": true,
                    "eventSource": true
                }
            }
        }
    },
    "organizations": {
        "ORG2MSP": {
            "mspid": "ORG2MSP",
            "peers": [
                "peer1.peers.org2.com",
                "peer2.peers.org2.com"
            ]
        }
    },
    "orderers": {
        "ord1.orderers.org1.com": {
            "url": "grpcs://10.100.xxx.xxx:7050",
            "grpcOptions": {
                "ssl-target-name-override": "ord1.orderers.org1.com",
                "request-timeout": 12000
            },
            "tlsCACerts": {
                "path": "temp.pem"
            }
        },
        "ord2.orderers.org1.com": {
            "url": "grpcs://10.100.xxx.xxx:7050",
            "grpcOptions": {
                "ssl-target-name-override": "ord2.orderers.org1.com",
                "request-timeout": 12000
            },
            "tlsCACerts": {
                "path": "temp.pem"
            }
        },
        "ord3.orderers.org1.com": {
            "url": "grpcs://10.100.xxx.xxx:7050",
            "grpcOptions": {
                "ssl-target-name-override": "ord3.orderers.org1.com",
                "request-timeout": 12000
            },
            "tlsCACerts": {
                "path": "temp.pem"
            }
        }
    },
    "peers": {
        "peer1.peers.org1.com": {
            "url": "grpcs://10.100.xxx.xxx:7051",
            "grpcOptions": {
                "ssl-target-name-override": "peer1.peers.org1.com",
                "request-timeout": 12000,
                "grpc.keepalive_time_ms": 600000
            },
            "tlsCACerts": {
                "path": "temp.pem"
            }
        },
        "peer1.peers.org3.com": {
            "url": "grpcs://public.ip.address:7051",
            "grpcOptions": {
                "ssl-target-name-override": "peer1.peers.org3.com",
                "request-timeout": 12000,
                "grpc.keepalive_time_ms": 600000
            },
            "tlsCACerts": {
                "path": "temp.pem"
            }
        },
        "peer1.peers.org2.com": {
            "url": "grpcs://10.100.xxx.xxx:7051",
            "grpcOptions": {
                "ssl-target-name-override": "peer1.peers.org2.com",
                "request-timeout": 12000,
                "grpc.keepalive_time_ms": 600000
            },
            "tlsCACerts": {
                "path": "temp.pem"
            }
        }
    }
}

Here is my code

'use strict';

const { Wallets, Gateway } = require('fabric-network');
const fs = require('fs');
const path = require('path');

const ccpPath = path.resolve(__dirname,'connection.json');
const ccpJSON = fs.readFileSync(ccpPath, 'utf8');
const ccp = JSON.parse(ccpJSON);


async function main(){
    try {
        // const walletPath = path.join(process.cwd(), 'wallet');
        const wallet = await Wallets.newFileSystemWallet('wallet');
        // console.log(`Wallet path: ${walletPath}`);

        // Check to see if we've already enrolled the user.
        const userExists = await wallet.get('usernew');
        const tlsExists = await wallet.get('tlsid');
        if (!userExists) {
            console.log('An identity for the user "usernew" does not exist in the wallet');
            return;
        }
        if (!tlsExists) {
            console.log('An identity for the user "tls" does not exist in the wallet');
            return;
        }
        console.log("Here");
        // Create a new gateway for connecting to our peer node.
        const gateway = new Gateway();
        await gateway.connect(ccp, { wallet, identity: 'usernew', discovery: { enabled: false, asLocalhost: false }, clientTlsIdentity: 'tlsid' });

        console.log("Here1");
        // Get the network (channel) our contract is deployed to.
        const network = await gateway.getNetwork('supplychain');

        console.log("Here2");
        //Get the channel object to fetch out peers
        const channel = network.getChannel();

        console.log("Here3");
        //Get peers for endorsement
        //channel.getEndorsers();
        const org1Peer = channel.getPeer('peer1.peers.org1.com');
        //console.log(org1Peer);
        const org2Peer = channel.getPeer('peer1.peers.org2.com');
        //console.log(org2Peer);
        const org3Peer = channel.getPeer('peer1.peers.org3.com');
        //console.log(org3Peer);
        // All the above logs print correct information


        // Get the contract from the network.
        const contract = network.getContract('mycontract');

        const result = await contract.evaluateTransaction('queryAllObjects');
        
        console.log(`Transaction has been evaluated, result is: ${result.toString()}`);

    } catch (error) {
        console.error(`Failed to evaluate transaction: ${error}`);
    }
}

main()

Here is the crypto folder tree

C:.
├───peers.org1.com
│   └───users
│       ├───Admin@peers.org1.com
│       │   ├───msp
│       │   │   ├───admincerts
│       │   │   ├───cacerts
│       │   │   ├───intermediatecerts
│       │   │   ├───keystore
│       │   │   ├───signcerts
│       │   │   ├───tlscacerts
│       │   │   └───tlsintermediatecerts
│       │   └───tls
│       └───User1@peers.org1.com
│           ├───msp
│           │   ├───admincerts
│           │   ├───cacerts
│           │   ├───intermediatecerts
│           │   ├───keystore
│           │   ├───signcerts
│           │   ├───tlscacerts
│           │   └───tlsintermediatecerts
│           └───tls
├───peers.org2.com
│   └───users
│       ├───Admin@peers.org2.com
│       │   ├───msp
│       │   │   ├───admincerts
│       │   │   ├───cacerts
│       │   │   ├───intermediatecerts
│       │   │   ├───keystore
│       │   │   ├───signcerts
│       │   │   ├───tlscacerts
│       │   │   └───tlsintermediatecerts
│       │   └───tls
│       └───User1@peers.org2.com
│           ├───msp
│           │   ├───admincerts
│           │   ├───cacerts
│           │   ├───intermediatecerts
│           │   ├───keystore
│           │   ├───signcerts
│           │   ├───tlscacerts
│           │   └───tlsintermediatecerts
│           └───tls
└───peers.org3.com
    └───users
        ├───Admin@peers.org3.com
        │   ├───msp
        │   │   ├───admincerts
        │   │   ├───cacerts
        │   │   ├───intermediatecerts
        │   │   ├───keystore
        │   │   ├───signcerts
        │   │   ├───tlscacerts
        │   │   └───tlsintermediatecerts
        │   └───tls
        └───User1@peers.org3.com
            ├───msp
            │   ├───admincerts
            │   ├───cacerts
            │   ├───intermediatecerts
            │   ├───keystore
            │   ├───signcerts
            │   ├───tlscacerts
            │   └───tlsintermediatecerts
            └───tls

The temp.pem used in the connection file above is prepared by appending the ica.pem and ca.pem shown below. Here is how the cerificates look for Org2. Looks similar for other 2 orgs. msp/tlscacerts/ca.pem

Issuer: C=XX, ST=XXXX, L=XXXX, O=MyCompany, OU=Cybersecurity, CN=MyCompany Root Certificate Authority 2018
Validity
    Not Before: Jul 23 17:07:45 2018 GMT
    Not After : Jul 23 17:17:44 2043 GMT
Subject: C=XX, ST=XXXX, L=XXXX, O=MyCompany, OU=Cybersecurity, CN=MyCompany Root Certificate Authority

msp/tlsintermediatecerts/ica.pem

Issuer: C=XX, ST=XXXX, L=XXXX, O=MyCompany, OU=Cybersecurity, CN=MyCompany Root Certificate Authority 2018
Validity
    Not Before: Nov 14 21:26:35 2018 GMT
    Not After : Nov 14 21:36:35 2025 GMT
Subject: C=XX, ST=XXXX, L=XXXX, O=MyCompany, CN=MyCompany Issuing CA 101

tls/server.crt

Issuer: C=XX, ST=XXXX, L=XXXX, O=MyCompany, CN=MyCompany Issuing CA 101
Validity
    Not Before: Jan 18 20:30:30 2021 GMT
    Not After : Jan 18 20:30:30 2023 GMT
Subject: C=XX, ST=XXXX, L=XXXX, O=MyCompany Inc., OU=org2client, CN=*.peers.org2.com
.
.
.
X509v3 Subject Alternative Name:
    DNS:*.peers.org2.com

Org2 NodeJs log

2021-02-25T10:21:33.736Z - error: [Endorser]: sendProposal[peer1.peers.org2.com] - Received error response from: grpcs://10.100.xxx.xxx:7051 error: Error: 2 UNKNOWN: error validating proposal: access denied: channel [supplychain] creator org [ORG2MSP]
2021-02-25T10:21:33.738Z - error: [Endorser]: sendProposal[peer1.peers.org2.com] - rejecting with: Error: 2 UNKNOWN: error validating proposal: access denied: channel [supplychain] creator org [ORG2MSP]
2021-02-25T10:21:33.738Z - error: [SingleQueryHandler]: evaluate: message=Query failed. Errors: ["Error: 2 UNKNOWN: error validating proposal: access denied: channel [supplychain] creator org [ORG2MSP]"], stack=FabricError: Query failed. Errors: ["Error: 2 UNKNOWN: error validating proposal: access denied: channel [supplychain] creator org [ORG2MSP]"]
    at SingleQueryHandler.evaluate (/fabric23/node_modules/fabric-network/lib/impl/query/singlequeryhandler.js:47:23)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Transaction.evaluate (/fabric23/node_modules/fabric-network/lib/transaction.js:276:25)
    at async main (/fabric23/test.js:67:25), name=FabricError
Failed to evaluate transaction: FabricError: Query failed. Errors: ["Error: 2 UNKNOWN: error validating proposal: access denied: channel [supplychain] creator org [ORG2MSP]"]

Org2 peer logs

2021-02-25 10:21:33.732 UTC [endorser] Validate -> WARN 08f access denied: creator's signature over the proposal is not valid: The signature is invalid channel=supplychain txID=01bde838 mspID=ORG2MSP
2021-02-25 10:21:33.732 UTC [comm.grpc.server] 1 -> INFO 090 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.23.238.200:40928 grpc.peer_subject="CN=*.peers.org3.com,OU=org3client,O=MyCompany Inc.,L=XXXX,ST=XXXX,C=XX" error="error validating proposal: access denied: channel [supplychain] creator org [ORG2MSP]" grpc.code=Unknown grpc.call_duration=12.335491ms

Org3 peer logs

2021-02-26 13:42:26.081 UTC [gossip.channel] publishStateInfo -> DEBU 6155d8 Empty membership, no one to publish state info to
2021-02-26 13:42:26.493 UTC [core.comm] ServerHandshake -> DEBU 6155d9 Server TLS handshake completed in 49.605106ms server=PeerServer remoteaddress=public.ip.address:291542021-02-26 13:42:26.597 UTC [grpc] InfoDepth -> DEBU 6155da [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2021-02-26 13:42:26.927 UTC [gossip.channel] publishStateInfo -> DEBU 6155db Empty membership, no one to publish state info to

I have also tried deploying the same code on the docker swarm on the azure vm. But there it gives the same error what I was getting when I was using the wrong certificate as given in the SO post here

Answer 1

Some points you may check:

• the peer server TLS cert of your org3 should have an alternative name like "*.ip.adress"?
• the channel has all 3 orgs right? From logs of org2 I see "creator's signature over the proposal is not valid"
• Check the user identity "usernew" PKI (not the TLS) to make sure the CA that issued the cert is one of the CA MSP on the channel. If you use intermediate CA those CAs certs should be also on the channel.

Best Regards, Tsvetan