无法在Hyperledger Fabric上查询或执行事务

Question

I've created a network which consists of 2 org having two peers each and a solo orderer. It was working fine before load testing after that the orderer went down, but I was still able to retrieve data from the peers. After a while, I restarted the network but now I am unable to query or perform a transaction.

Whenever I am trying to query from cli I am getting:

Error: error getting endorser client for query: endorser client failed to connect to peer0.blockit.gatelesis.com:7051: failed to create new connection: context deadline exceeded

When I am trying to query from the peer directly I got:

Error: error endorsing query: rpc error: code = Unknown desc = Failed evaluating policy on signed data during check policy on channel [channel-24] with policy [/Channel/Application/Writers]: [implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied] - proposal response: <nil>

At peer logs I got this:

2019-08-28 10:00:49.059 UTC [nodeCmd] registerDiscoveryService -> INFO a33d Discovery service activated
2019-08-28 10:00:49.060 UTC [nodeCmd] serve -> INFO a33e Starting peer with ID=[name:"peer0.blockit.gatelesis.com" ], network ID=[dev], address=[peer0.blockit.gatelesis.com:7051]
2019-08-28 10:00:49.060 UTC [nodeCmd] serve -> INFO a33f Started peer with ID=[name:"peer0.blockit.gatelesis.com" ], network ID=[dev], address=[peer0.blockit.gatelesis.com:7051]
2019-08-28 10:00:49.061 UTC [core.comm] ServerHandshake -> ERRO a340 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:53884
2019-08-28 10:00:49.061 UTC [core.comm] ServerHandshake -> ERRO a341 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:40656
2019-08-28 10:00:49.061 UTC [core.comm] ServerHandshake -> ERRO a342 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:46028
2019-08-28 10:00:49.061 UTC [core.comm] ServerHandshake -> ERRO a343 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:49532
2019-08-28 10:00:49.062 UTC [core.comm] ServerHandshake -> ERRO a344 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:50544
2019-08-28 10:00:49.062 UTC [core.comm] ServerHandshake -> ERRO a345 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:50950
2019-08-28 10:00:49.062 UTC [core.comm] ServerHandshake -> ERRO a346 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:51294
2019-08-28 10:00:49.063 UTC [core.comm] ServerHandshake -> ERRO a347 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:51580
2019-08-28 10:00:49.063 UTC [nodeCmd] func8 -> INFO a348 Starting profiling server with listenAddress = 0.0.0.0:6060
2019-08-28 10:00:49.063 UTC [core.comm] ServerHandshake -> ERRO a349 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:51854
2019-08-28 10:00:49.063 UTC [core.comm] ServerHandshake -> ERRO a34a TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:53742
2019-08-28 10:00:49.064 UTC [core.comm] ServerHandshake -> ERRO a34b TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:52088
2019-08-28 10:00:49.064 UTC [core.comm] ServerHandshake -> ERRO a34c TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:52316
2019-08-28 10:00:49.064 UTC [core.comm] ServerHandshake -> ERRO a34d TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.5:43472
2019-08-28 10:00:49.064 UTC [core.comm] ServerHandshake -> ERRO a34e TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:52524
2019-08-28 10:00:49.065 UTC [core.comm] ServerHandshake -> ERRO a34f TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:53094
2019-08-28 10:00:49.065 UTC [core.comm] ServerHandshake -> ERRO a350 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:53260
2019-08-28 10:00:49.065 UTC [core.comm] ServerHandshake -> ERRO a351 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:53428
2019-08-28 10:00:49.065 UTC [core.comm] ServerHandshake -> ERRO a352 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:53580
2019-08-28 10:00:49.065 UTC [core.comm] ServerHandshake -> ERRO a353 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:52724
2019-08-28 10:00:49.066 UTC [core.comm] ServerHandshake -> ERRO a354 TLS handshake failed with error EOF server=PeerServer remoteaddress=172.25.0.3:52908
2019-08-28 10:00:49.709 UTC [gossip.election] beLeader -> INFO a355 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:49.709 UTC [gossip.service] func1 -> INFO a356 Elected as a leader, starting delivery service for channel channel-988
2019-08-28 10:00:50.459 UTC [gossip.election] beLeader -> INFO a357 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:50.459 UTC [gossip.service] func1 -> INFO a358 Elected as a leader, starting delivery service for channel channel-989
2019-08-28 10:00:50.895 UTC [gossip.election] beLeader -> INFO a359 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:50.895 UTC [gossip.service] func1 -> INFO a35a Elected as a leader, starting delivery service for channel channel-99
2019-08-28 10:00:51.251 UTC [gossip.election] beLeader -> INFO a35b dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:51.251 UTC [gossip.service] func1 -> INFO a35c Elected as a leader, starting delivery service for channel channel-990
2019-08-28 10:00:51.600 UTC [gossip.election] beLeader -> INFO a35d dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:51.601 UTC [gossip.service] func1 -> INFO a35e Elected as a leader, starting delivery service for channel channel-991
2019-08-28 10:00:51.950 UTC [gossip.election] beLeader -> INFO a35f dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:51.950 UTC [gossip.service] func1 -> INFO a360 Elected as a leader, starting delivery service for channel channel-992
2019-08-28 10:00:52.017 UTC [comm.grpc.server] 1 -> INFO a361 unary call completed grpc.service=gossip.Gossip grpc.method=Ping grpc.request_deadline=2019-08-28T10:00:54.017Z grpc.peer_address=172.25.0.3:54020 grpc.peer_subject="CN=peer1.blockit.gatelesis.com,L=San Francisco,ST=California,C=US" grpc.code=OK grpc.call_duration=91.044µs
2019-08-28 10:00:52.020 UTC [comm.grpc.server] 1 -> INFO a362 streaming call completed grpc.service=gossip.Gossip grpc.method=GossipStream grpc.request_deadline=2019-08-28T10:01:02.018Z grpc.peer_address=172.25.0.3:54020 grpc.peer_subject="CN=peer1.blockit.gatelesis.com,L=San Francisco,ST=California,C=US" error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=1.8267ms
2019-08-28 10:00:52.051 UTC [comm.grpc.server] 1 -> INFO a363 unary call completed grpc.service=gossip.Gossip grpc.method=Ping grpc.request_deadline=2019-08-28T10:00:54.051Z grpc.peer_address=172.25.0.3:54022 grpc.peer_subject="CN=peer1.blockit.gatelesis.com,L=San Francisco,ST=California,C=US" grpc.code=OK grpc.call_duration=79.786µs
2019-08-28 10:00:52.308 UTC [gossip.election] beLeader -> INFO a364 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:52.308 UTC [gossip.service] func1 -> INFO a365 Elected as a leader, starting delivery service for channel channel-993
2019-08-28 10:00:52.338 UTC [comm.grpc.server] 1 -> INFO a366 streaming call completed grpc.service=gossip.Gossip grpc.method=GossipStream grpc.peer_address=172.25.0.3:54022 grpc.peer_subject="CN=peer1.blockit.gatelesis.com,L=San Francisco,ST=California,C=US" error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=285.951264ms
2019-08-28 10:00:52.722 UTC [gossip.election] beLeader -> INFO a367 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:52.722 UTC [gossip.service] func1 -> INFO a368 Elected as a leader, starting delivery service for channel channel-994
2019-08-28 10:00:52.853 UTC [gossip.comm] func1 -> WARN a369 peer1.blockit.gatelesis.com:8051, PKIid:b6112b61b1885609860b91b49516a61a9327cb1b82f79b1c2d09e13a4c30a924 isn't responsive: EOF
2019-08-28 10:00:52.853 UTC [gossip.discovery] expireDeadMembers -> WARN a36a Entering [b6112b61b1885609860b91b49516a61a9327cb1b82f79b1c2d09e13a4c30a924]
2019-08-28 10:00:52.853 UTC [gossip.discovery] expireDeadMembers -> WARN a36b Closing connection to Endpoint: peer1.blockit.gatelesis.com:8051, InternalEndpoint: peer1.blockit.gatelesis.com:8051, PKI-ID: b6112b61b1885609860b91b49516a61a9327cb1b82f79b1c2d09e13a4c30a924, Metadata:
2019-08-28 10:00:52.853 UTC [gossip.discovery] expireDeadMembers -> WARN a36c Exiting
2019-08-28 10:00:52.952 UTC [gossip.election] beLeader -> INFO a36d dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:52.952 UTC [gossip.service] func1 -> INFO a36e Elected as a leader, starting delivery service for channel channel-995
2019-08-28 10:00:54.029 UTC [gossip.election] beLeader -> INFO a36f dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:54.029 UTC [gossip.service] func1 -> INFO a370 Elected as a leader, starting delivery service for channel channel-996
2019-08-28 10:00:54.378 UTC [gossip.election] beLeader -> INFO a371 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:54.378 UTC [gossip.service] func1 -> INFO a372 Elected as a leader, starting delivery service for channel channel-997
2019-08-28 10:00:54.717 UTC [gossip.election] beLeader -> INFO a373 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:54.717 UTC [gossip.service] func1 -> INFO a374 Elected as a leader, starting delivery service for channel channel-998
2019-08-28 10:00:55.058 UTC [gossip.election] beLeader -> INFO a375 dfc797622a9f875949601983ffceb8b5e48951174366174209299611544d92bf : Becoming a leader
2019-08-28 10:00:55.058 UTC [gossip.service] func1 -> INFO a376 Elected as a leader, starting delivery service for channel channel-999
2019-08-28 10:15:18.931 UTC [comm.grpc.server] 1 -> INFO a377 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.25.0.5:44006 error="Failed evaluating policy on signed data during check policy on channel [channel-24] with policy [/Channel/Application/Writers]: [implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied]" grpc.code=Unknown grpc.call_duration=5.935604ms
2019-08-28 10:15:52.432 UTC [comm.grpc.server] 1 -> INFO a378 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.25.0.5:44014 error="Failed evaluating policy on signed data during check policy on channel [channel-24] with policy [/Channel/Application/Writers]: [implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied]" grpc.code=Unknown grpc.call_duration=507.807µs
2019-08-28 10:18:49.674 UTC [comm.grpc.server] 1 -> INFO a379 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.25.0.5:44076 error="Failed evaluating policy on signed data during check policy on channel [channel-24] with policy [/Channel/Application/Writers]: [implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied]" grpc.code=Unknown grpc.call_duration=439.412µs

At orderer logs I got this:

2019-08-28 09:52:31.880 UTC [orderer.common.cluster] loadVerifier -> INFO a74 Loaded verifier for channel paradox from config block at index 2673
2019-08-28 09:52:31.881 UTC [orderer.common.server] initializeServerConfig -> INFO a75 Starting orderer with TLS enabled
2019-08-28 09:52:31.900 UTC [orderer.common.server] initializeMultichannelRegistrar -> INFO a76 Not bootstrapping because of existing chains
2019-08-28 09:52:52.091 UTC [orderer.commmon.multichannel] Initialize -> INFO a77 Starting system channel 'paradox' with genesis block hash d771e0c05b145ce05e549e379a4804032833fe2b292cf2b147da975921866ae4 and orderer type solo
2019-08-28 09:52:52.091 UTC [orderer.common.server] Start -> INFO a78 Starting orderer:
Version: 1.4.1
Commit SHA: 87074a7
Go version: go1.11.5
OS/Arch: linux/amd64
2019-08-28 09:52:52.091 UTC [orderer.common.server] Start -> INFO a79 Beginning to serve requests
2019-08-28 09:52:52.093 UTC [core.comm] ServerHandshake -> ERRO a7a TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55816
2019-08-28 09:52:52.101 UTC [core.comm] ServerHandshake -> ERRO a7b TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55686
2019-08-28 09:52:52.102 UTC [core.comm] ServerHandshake -> ERRO a7c TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55560
2019-08-28 09:52:52.102 UTC [core.comm] ServerHandshake -> ERRO a7d TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55562
2019-08-28 09:52:52.103 UTC [core.comm] ServerHandshake -> ERRO a7e TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55564
2019-08-28 09:52:52.103 UTC [core.comm] ServerHandshake -> ERRO a7f TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55566
2019-08-28 09:52:52.103 UTC [core.comm] ServerHandshake -> ERRO a80 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55568
2019-08-28 09:52:52.104 UTC [core.comm] ServerHandshake -> ERRO a81 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55570
2019-08-28 09:52:52.104 UTC [core.comm] ServerHandshake -> ERRO a82 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55572
2019-08-28 09:52:52.105 UTC [core.comm] ServerHandshake -> ERRO a83 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55574
2019-08-28 09:52:52.108 UTC [core.comm] ServerHandshake -> ERRO a84 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55576
2019-08-28 09:52:52.108 UTC [core.comm] ServerHandshake -> ERRO a85 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55578
2019-08-28 09:52:52.109 UTC [core.comm] ServerHandshake -> ERRO a86 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55580
2019-08-28 09:52:52.109 UTC [core.comm] ServerHandshake -> ERRO a87 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55582
2019-08-28 09:52:52.110 UTC [core.comm] ServerHandshake -> ERRO a88 TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55584

All the nodes are running in docker containers. Any help will be appreciated.

I've attached extended logs for the orderer and peer

Peer: https://pastebin.com/DP0gr56h

Orderer: https://pastebin.com/GYnEXyax

Answer 1

2019-08-28 09:52:52.093 UTC [core.comm] ServerHandshake -> ERRO a7a TLS handshake failed with error EOF server=Orderer remoteaddress=172.25.0.5:55816

When I carefully observed both Orderer and peer logs I saw below error

TLS handshake failed with error EOF >>> Makes me to think about status of the Orderer and peer

Both trying to connect each other and not able to connect