Can a lock-free atomic write / consistent read operation be achieved on a 4 byte int using System V shared memory across language platforms?

Question

I want to implement a lock free counter , a 4-byte int , in System V shared memory. The writer is a C++ program, the reader is a Python program. Working roughly like this:

• C++ code updates counter in an atomic operation
• Python code reads counter and has a consistent view of memory (eventual consistency is perfectly acceptable)
• No locks are implemented to achieve this

Within the C++ language there are atomic get/update operations that allow for this and guarantee memory consistency, I believe the same is true in Python.

However, as I understand it, the assumptions in C++ regarding atomic operations do not necessarily apply to code written and compiled in another language and compiler.

Is there a way to achieve a consistent view of shared memory across languages that doesn't involve implementing low level locks?

Answer 1

Is there a way to achieve a consistent view of shared memory across languages that doesn't involve implementing low level locks?

No, not in general.

First I would say this has nothing to do with languages but more with the actual platforms, architecture, implementation or operating system.

Because languages differ quite strongly, take Python for example: It has no language native way of accessing memory directly or lets say in an low level manner. However some of its implementations do offer its own API. Languages intended for such low level use have abstractions for that, as C, C++ or Rust have. But that abstractions are then implemented often quite differently, as they often depend on where the code is run, interpreted or compiled for. An integer for some architectures are big endian, on most, like x86 or arm, its little endian. Operating systems also have a say, as for example memory is used and abstracted.

And while many languages have a common abstractions of linear memory, its gets even messier with atomics: The compiler for C++ could generate machine code ie assembly that check if the CPU run on does support new fancy atomic integer instructions and use them or fall back on often supported atomic flags plus the integer. It could just rely on operating system, a spin lock, or standardized APIs defined by POSIX, SystemV, Linux, Windows if the code has the luxury of running in an operating system managed environment .

For non imperative languages it gets even messier.

So in order to exchange data between languages, the implementations of those languages have to use some common exchange. They could try to do that directly via shared memory for instance. This is then called an Application Binary Interface (ABI), as a memory abstraction is at least common to them a priori. Or the operating system or architecture might even standardized such things or even supports APIs.

System V would be an API designed for such interchange but since AFAIK it does not have an abstraction for atomics or lock less abstractions, the answer stays no, even with the System V context out of the title.

Answer 2

I'm going to take issue with some of Superlokkus' assertions.

The mmap primitive is available in both C++ and Python. That primitive gives you memory that is in a physical page shared by both processes. Different virtual addresses, same physical page of memory. Changes by one process are immediately viewable by the other process. That's how it has to work. It operates at a hardware level. Abstractions are irrelevant.

Now, that DOESN'T mean you can get notification of those changes. If you are polling in a loop (presumably a friendly loop with sleeping in between checks), then you will see the change the next time you check.

Answer 3

Yes, using the atomics library, along with a suitable shared memory library (eg mmap or shared_memory ).
This example assumes your atomic int is in the first 4 bytes of the shared memory segment.

from atomics import atomicview, MemoryOrder, INT
from multiprocessing import shared_memory


# connect to existing shared memory segment
shmem = SharedMemory(name="test_shmem")

# get buf corresponding to "atomic" region
buf = shmem.buf[:4]

# atomically read from buffer
with atomicview(buffer=buf, atype=INT) as a:
    value = a.load(order=MemoryOrder.ACQUIRE)

# print our value
print(value)

# del our buf object (or shmem.close() will complain)
del buf

# close our shared memory handle
shmem.close()

We can use ACQUIRE memory order here rather than the default SEQ_CST .

The atomicview can only be created and used with a with statement, so you will need to manually keep your buf around (and manage its lifetime correctly).

Note: I am the author of this library