Realtime data sharing in between threads

Question

For an embedded systems project, I have to deal with sensors that send a lot of data concurrently. At the moment, each sensor has its own thread and many threads have references to each other.

Using mutex locking, these threads obtain data from each other. However during production some threads wait indefinitely on another thread to finish working with locked data. I know that the issue has to do with deadlocking, but I find these problems hard to find and to prevent.

I want to avoid using mutexes this extensively, since they cause most of my hard to reproduce problems. I have tried many things, such as automatically unlocking a mutex when it goes out of scope, but nothing has worked so far. I have a SharedData class which contains the following methods:

```
template<class T>
T SharedData<T>::Get() {
  LockGuard lock(mutex_);
  T data = data_;
  if (!IsValid() && has_default_value_) {
    data = default_value_;
  }

  return data;
}

template<class T>
void SharedData<T>::Set(T data) {
  is_set_ = true;
  set_time_ = system_clock::now();

  LockGuard lock(mutex_);
  data_ = data;
}
```

My question is as follows; what is a good, safe way to share realtime data between threads (preferably without using mutexes)?

I am looking for a solution in the direction of message passing in between threads. I have not yet found an elegant way to do this.

Thanks in advance!

Edit: to clarify 'threads obtaining data from each other', here is a code snippet:

void MotorMessage::SetConnectedModules(MotorSensor &motor_sensor) {
  out_buffer_[index_++] = motor_sensor.connected_.Get();
}

Here motor_sensor is a reference to a different thread, and connected_ is the SharedData type.

Answer 1

You can set up one or multiple atomic queues from the sensor threads to the consumer(s). This way you don't have to do any locking on your own.

For example, a queue from Intel TBB .

Answer 2

If you design your program in such a way that no thread ever attempts to lock a second mutex while already holding a first locked mutex, then your program will be guaranteed never to deadlock.

If you can't manage that, you can still guarantee that your program will never deadlock if you ensure that whenever any thread attempts to lock more than one mutex at a time, it attempts the lock those two (or more) mutexes in the same order that every other thread attempts to lock them in. (ie it's only when thread #1 does a lock(mutex_a); lock(mutex_b); while thread #2 does a lock(mutex_b); lock(mutex_a); that you open up the possibility that a deadlock can -- and therefore eventually will -- occur)

As for how to do this, if your program is small enough that it's still practical to redesign it, then a good design to use is that of message-passing between threads rather than shared-data. That is, if thread A has some data that thread B needs to know about, thread A should wrap that data up in some sort of message/event object and post that object to an message/event-queue that thread B will be notified about and check for later. (Since neither A's posting of the event nor B's receiving of the event will ever block for more than a small/finite amount of time, all possibility of deadlocks is avoided by this approach) Note that even if the amount of data you want to pass across threads is large, this technique will still be efficient if you are passing the data via shared_ptr rather than by making a copy of the data.

If, OTOH, your program is already too large/complicated to make a redesign feasible, your other option is to analyze and debug why your program is deadlocking, and come up with the necessary changes to avoid those deadlocks. Tools like valgrind's helgrind can help with this, by automatically detecting inconsistent-mutex-locking at runtime and telling you about it. Also, if you can get your program into a deadlocked state, a debugger can show you where each thread is blocked at, and that can lead you towards understanding where in your codebase the inconsistent locking-order is that is allowing the deadlock to occur.

Short of that, you can always instrument each lock/unlock command with a printf() that includes both the thread ID and a unique identifier for the mutex (eg its memory-location) and then go over the resulting log-output to search for inconsistent locking-patterns in the output. This log-parsing can be automated with a tool if it becomes too difficult to do it manually.