DNS resolver giving wrong IP address when running Hyperledger Fabric Gateway

Question

I am running a JS file which acts as a listener to a set of channels on a Hyperledger Fabric network. In spite of the following error I am describing, the listener is able to listen to the channel (which puzzles me) based on evidence in the console output (an excerpt of which is provided below). As this code is going into production, I need to resolve whatever errors there are, even if the functionality is not affected.

So I have a problem with the following JavaScript code excerpt:

let network, listener;
for (var i = 0; i < channelid.length; i++) {
  let cid = channelid[i];
  console.log(`Adding listener for channel ${cid}`);
  network = await gateway.getNetwork(cid);
  console.log(`await gateway.getNetwork`);
  listener = await network.addBlockListener(
    async (event) => {
      await ProcessingMap.set(event.blockData.header.number, event.blockData);
      console.log(
        `Added block ${event.blockData.header.number} to ProcessingMap`,
      );
    },
    {startBlock: parseInt(nextBlock, 10)},
  );
  console.log(`await network.addBlockListener`);
}

(Full code can be found at https://pastebin.com/qqFPdVH6 )

When I run this code with export GRPC_VERBOSITY=DEBUG and export GRPC_TRACE=all , I get the following error at the line network = await gateway.getNetwork(cid) (the full log up to when the blocks start getting read can be found at https://pastebin.com/ZxJBx4ub )

2020-10-06T07:43:24.424Z | resolving_load_balancer | dns:orderer1.isprint.com:7050 IDLE -> IDLE
2020-10-06T07:43:24.424Z | connectivity_state | dns:orderer1.isprint.com:7050 IDLE -> IDLE
2020-10-06T07:43:24.424Z | dns_resolver | Resolver constructed for target dns:orderer1.isprint.com:7050
2020-10-06T07:43:24.425Z | dns_resolver | Resolution update requested for target dns:orderer1.isprint.com:7050
2020-10-06T07:43:24.425Z | resolving_load_balancer | dns:orderer1.isprint.com:7050 IDLE -> CONNECTING
2020-10-06T07:43:24.425Z | connectivity_state | dns:orderer1.isprint.com:7050 IDLE -> CONNECTING
2020-10-06T07:43:24.426Z | resolving_load_balancer | dns:orderer1.isprint.com:7050 CONNECTING -> CONNECTING
2020-10-06T07:43:24.426Z | connectivity_state | dns:orderer1.isprint.com:7050 CONNECTING -> CONNECTING
2020-10-06T07:43:24.427Z | dns_resolver | Resolved addresses for target dns:orderer1.isprint.com:7050: [35.186.238.101:7050]
2020-10-06T07:43:24.427Z | pick_first | IDLE -> IDLE
2020-10-06T07:43:24.427Z | resolving_load_balancer | dns:orderer1.isprint.com:7050 CONNECTING -> IDLE
2020-10-06T07:43:24.427Z | connectivity_state | dns:orderer1.isprint.com:7050 CONNECTING -> IDLE
2020-10-06T07:43:24.427Z | pick_first | Connect to address list 35.186.238.101:7050
2020-10-06T07:43:24.427Z | subchannel | 35.186.238.101:7050 refcount 0 -> 1
2020-10-06T07:43:24.427Z | subchannel | 35.186.238.101:7050 refcount 1 -> 2
2020-10-06T07:43:24.427Z | pick_first | Start connecting to subchannel with address 35.186.238.101:7050
2020-10-06T07:43:24.427Z | pick_first | IDLE -> CONNECTING
2020-10-06T07:43:24.427Z | resolving_load_balancer | dns:orderer1.isprint.com:7050 IDLE -> CONNECTING
2020-10-06T07:43:24.427Z | connectivity_state | dns:orderer1.isprint.com:7050 IDLE -> CONNECTING
2020-10-06T07:43:24.427Z | subchannel | 35.186.238.101:7050 IDLE -> CONNECTING
2020-10-06T07:43:24.427Z | pick_first | CONNECTING -> CONNECTING
2020-10-06T07:43:24.427Z | resolving_load_balancer | dns:orderer1.isprint.com:7050 CONNECTING -> CONNECTING
2020-10-06T07:43:24.427Z | connectivity_state | dns:orderer1.isprint.com:7050 CONNECTING -> CONNECTING
2020-10-06T07:43:27.428Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Committer- name: orderer1.isprint.com:7050, url:grpcs://orderer1.isprint.com:7050, connected:false, connectAttempted:true
2020-10-06T07:43:27.429Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server orderer1.isprint.com:7050 url:grpcs://orderer1.isprint.com:7050 timeout:3000
2020-10-06T07:43:27.429Z - error: [DiscoveryService]: _buildOrderer[isprintchannel] - Unable to connect to the discovered orderer orderer1.isprint.com:7050 due to Error: Failed to connect before the deadline on Committer- name: orderer1.isprint.com:7050, url:grpcs://orderer1.isprint.com:7050, connected:false, connectAttempted:true

I have no idea how come the DNS resolver is giving me that weird IP address 35.186.238.101, so I did nslookup 35.186.238.101 :

[root@isprintdev nodejs]# nslookup 35.186.238.101
Server:         192.168.1.8
Address:        192.168.1.8#53

Non-authoritative answer:
101.238.186.35.in-addr.arpa     name = 101.238.186.35.bc.googleusercontent.com.

Authoritative answers can be found from:

[root@isprintdev nodejs]#

The connection profile can be found at https://pastebin.com/Ppe9JPXR . In short, my network consists of three orderers and two peers, all on the same virtual machine.

So my question is : is there something that I need to change in my connection profile or JS code (or wherever necessary) to ensure that this error does not appear?

Answer 1

isprint.com has a public DNS entry and apparently a wildcard DNS A record which resolves *.isprint.com to 35.186.238.101

MacBook-Pro:~ garis$ nslookup orderer1.isprint.com
Server:     192.168.1.1
Address:    192.168.1.1#53

Non-authoritative answer:
Name:   orderer1.isprint.com
Address: 35.186.238.101

Not sure how you are running Fabric (Docker, binaries, etc) and not sure how you are running your client, but looks like your JS app is likely running directly on the host VM and is therefore using the configured DNS resolution which resolves as above. Assuming your Fabric nodes are running via Docker / Docker Compose, they can get away with using isprint.com as they'll be resolved on the Docker network prior to external DNS resolution.

So to save your problem, you'll need to create a HOSTS file entry which maps your endpoints to the IP within the VM that the client can use to access the Fabric nodes.

On *nix systems, you would need to edit etc/hosts and add an entry like:

192.168.50.65   orderer1.isprint.com    orderer2.isprint.com    orderer3.isprint.com