Synchronized code block for microservice architecture

Question

We have a method that needs to have synchronized for a code block across all instances of microservice.

MyEntity (class) in DynamoDB Example:

field1(Partition Key), field2(Sort Key), isPrimary
1                           1               true
1                           2               false
1                           3               false
2                           1               true
2                           2               false
3                           2               true

The value of one type based on field1 can be primary which gets decided based on field2.

Like, out of all 1s one will be primary, and so on for 2s and 3s in the field1 which is decided based on the value of the field2.

If the value of field1 is new in the entire table then based on FCFS logic it will become primary.

Problem: The problem is in a microservice architecture, if more than one instance is trying to insert having same value of field1 which does not exists in table then they might end up making both as primary. As we are using DynamoDB so cannot put constraint on table.

public MyEntity saveMyEntity(MyEntity myEntity) {
    if (!myEntity.isPrimary()) {
        synchronized (myEntity.getField1().intern()) {
            if (!existsByField1(myEntity.getField1())) {
                myEntity.setPrimary(true);
            }
        }
    }
    dynamoDBMapper.save(myEntity);
    return myEntity;
}

private boolean existsByField1(String field1){
    //checks if field1 value exists in table column field1.
}

Tried making block level synchronization but it will work only for the same instance not for others.

How can we achieve microservice-level code block synchronization?

Answer 1

What you are looking for is distributed locking: you use a strongly consistent linearizable datastore to manage locks. Examples of such stores include zookeeper, etcd, and consul. You can also implement it yourself by having the instances communicate with each other and implement a consensus protocol like Raft or Paxos (though it must be said that rolling one's own consensus protocol is in a similar category to rolling one's own cryptography: useful as a learning exercise, but probably not something you should want in production).

Note that the consistency demands here (I'm making the assumption that if there were multiple primaries that would be Really Bad) imply that there are unavoidable scenarios where your system will grind to a halt for at least some period of time (eg an instance of the service gets stopped or crashes while holding the locks). You can reduce that period of time, but doing so in general implies an increasing risk of multiple primaries, because there's no easy way to guarantee that the instance holding the lock is actually not doing anything with it.