Hazelcast-客户端模式拓扑/分布式地图锁定问题

Question

Below is the description of problem we faced in production. Please note that I could not reproduce the issue in test or local environment and therfore can not provide you with test code.

We have a hazelcast cluster with two members M1, M2 and three clients C1,C2,C3. Hazelcast version is 3.9.

Clients use IMap.tryLock() method with timeout of 10 seconds. After getting the lock, critical and long running operations are performed and finally the lock is released using IMap.unlock() method.

The problem occured in production is as follows:

At some time instant t, we first saw heartbeat failure to M2 at client C2. Afterwards there are errors in fetching partition table casued by com.hazelcast.spi.exception.TargetDisconnectedException:

[hz.client_0.internal-2                       ] WARN  [] HeartbeatManager               - hz.client_0 [mygroup] [3.9] HeartbeatManager failed to connection: .....

[hz.client_0.internal-3                       ] WARN  [] ClientPartitionService         - hz.client_0 [mygroup] [3.9] Error while fetching cluster partition table!
java.util.concurrent.ExecutionException: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to owner connection ClientConnection{alive=true, connectionId=1, ......

Around 250 ms after initial heartbeat failure, client gets disconnected and then reconnects in 20 ms.

[hz.client_0.cluster-                         ] INFO  [] LifecycleService               - hz.client_0 [mygroup] [3.9] HazelcastClient 3.9 (20171023 - b29f549) is CLIENT_DISCONNETED

[hz.client_0.cluster-                         ] INFO  [] LifecycleService               - hz.client_0 [mygroup] [3.9] HazelcastClient 3.9 (20171023 - b29f549) is CLIENT_CONNECTED

The problem we are having is, for some keys that are previously acquired by C2, C1 and C3 can not acquire the lock even if it seems to be released by C2. C2 can get the lock, but this puts unacceptable delays to the application and is not acceptable.. All clients should get since lock is released...

We were notified of the problem after receiving complaints, and then restarted the client application C2.

As documented in http://docs.hazelcast.org/docs/latest-development/manual/html/Distributed_Data_Structures/Lock.html , locks acquired by restarted member (C2 in my case) seemed to be removed after restart operation.

Currently the issue seems to go away, but we are not sure if it will recur.

Do you have any suggestions about the probable cause and more importantly do you have any recommendations?

Would enabling redo-operation in client help for this problem case?

As I tried to explain client seems to recover the problem, but keys remain locked in cluster and this is fatal to my application.

Thanks

Answer 1

It looks like the client had lost the ownership of the lock because of its disconnection from the cluster. You can use IMap#forceUnlock API in cases such as you faced. It releases the lock regardless of the lock owner and it always successfully unlocks, never blocks, and returns immediately.