pthread_mutex_lock / unlock的性能

Question

I've noticed that I take a pretty big performance hit when I have an algorithm that locks and unlocks a thread ALOT.

Is there any way to help this overhead? Would using a semaphore be more/less efficient?

Thanks

typedef struct _treenode{
   struct _treenode *leftNode;
   struct _treenode *rightNode;
   int32_t data;
   pthread_mutex_t mutex;
}TreeNode;

pthread_mutex_t _initMutex = PTHREAD_MUTEX_INITIALIZER;

int32_t insertNode(TreeNode **_trunk, int32_t data){
   TreeNode **current;
   pthread_mutex_t *parentMutex = NULL, *currentMutex = &_initMutex;

   if(_trunk != NULL){
      current = _trunk;
      while(*current != NULL){
         pthread_mutex_lock(&(*current)->mutex);
         currentMutex = &(*current)->mutex;
         if((*current)->data < data){
            if(parentMutex != NULL)
               pthread_mutex_unlock(parentMutex);
            pthreadMutex = currentMutex;
            current = &(*current)->rightNode;
         }else if((*current)->data > data){
            if(parentMutex != NULL)
               pthread_mutex_unlock(parentMutex);
            parentMutex = currentMutex;
            current = &(*current)->leftNode;
         }else{
            pthread_mutex_unlock(currentMutex);
            if(parentMutex != NULL)
               pthread_mutex_unlock(parentMutex);
            return 0;
         }
      }
      *current = malloc(sizeof(TreeNode));
      pthread_mutex_init(&(*current)->mutex, NULL);
      pthread_mutex_lock(&(*current)->mutex);
      (*current)->leftNode = NULL;
      (*current)->rightNode = NULL;
      (*current)->data = data;
      pthread_mutex_unlock(&(*current)->mutex);
      pthread_mutex_unlock(currentMutex);
   }else{
      return 1;
   }
   return 0;
}

int main(){
   int i;
   TreeNode *trunk = NULL;
   for(i=0; i<1000000; i++){
      insertNode(&trunk, rand() % 50000);
   }
}

Answer 1

Instead of worrying about the blades of grass, step back and observe the whole forest.

Any algorithm which depends on two threads potentially closely stepping on each each other's toes is inherently inefficient. Try to find a way to drastically reduce the need for interaction.

For example, if one thread produces data and the other consumes it, one can easily think up an inefficient algorithm where the producer publishes the data in shared memory and then waits for the other to consume it. Meanwhile the consumer is waiting for the producer to finish, etc., etc. This is all much simplified by the producer writing into a file or pipe, and the consumer reading from it.

Answer 2

pthread_mutex_lock and pthread_mutex_unlock vary in cost depending on contention:

1. Single thread use - either only one thread exists, or only one thread is using the mutex and the resource it protects: locking is virtually free , perhaps 80-100 cycles at most.
2. Multiple threads using the resource, but locks are held for very short intervals and contention is rare: locking has some cost, and it's hard to measure; the cost consists mostly of invalidating other cores'/cpus' cache lines.
3. Significant lock contention: nearly every lock and unlock operation will require assistance from the kernel, and the cost is easily several thousand (possibly even tens of thousand) cycles per lock/unlock.

Still, mutexes should be the least expensive locking primitive in most situations and on most implementations. Occasionally spinlocks may perform better. I would never expect semaphores to perform better.

Answer 3

As far as I can see your lock strategy is not optimal since most of the locks will not be taken to change the data, but just to read and find the way through the tree.

pthread_rwlock_t could help on this. You'd only take read-locks on the path down in the tree until you hit a node where you want to do some modification. There you would then take a write-lock. By that you could have other threads perform the same task when walking down the tree in a different branch without disturbing each other.

A decent implementation of pthread_rwlock_t would do this with a counter for the readers that it changes with atomic operations, as long as there is no contention with writers. This should be very fast. Once there is contention, it would be as costly as a mutex, I think.

Answer 4

Your locks are probably too fine-grained. Of course, the optimal granularity may vary depending on workload.

You could use a single lock for the whole tree, and it may perform better. But, if you do lots of reading and relatively few insertions/deletions, you end up with the whole tree locked often for no good reason. You may want to use a reader-writer lock, which would allow several readers at the same time.

Your question reminded me of this other one , when there is a comparison between fine-grained locking and coarse-grained locking for a linked list. While in the coarse-grained version each thread run in turn (not in parallel), and the total running time was slightly more than the sum of each thread's running time, and in the fine-grained version total running time was much less than the sum of each thread's running time, the added overhead of fine-grained locking totally offset these benefit, making the fine-grained version slower than the coarse-grained one.

Answer 5

Locking and unlocking are very expensive operations in the case of pthread_mutex_lock/unlock. With more details on the algorithm I could make some suggestions but as far as I can tell I can't tell you anything for certain. Semaphores are an alternative (again depending on the algorithm) and also barriers are another useful method for concurrency. To help the overhead you can do things like either make your locks smaller granularity or greater granularity. locks inside loops that iterate multiple times are a bad idea and you may want to move them outside the loop. This is just one example but there are probably more I can come up with. It's about determining whether the cost of the lock is greater than that of the critical section of your code. If you provide your algorithm or some sample code I'd be glad to take a look at it.

Answer 6

pthread_mutex_lock and pthread_cond_wait are OS primitives - they put calling thread into sleep, transferring control to another thread. Ie they involve syscalls and a lot of overhead. In tight integration between two threads, you don't really want to relinquish any control for even a cycle.

Give that I suggest using volatile int variables instead of mutexes:

volatile int data_ready = 0;
/*  ... */
while (!data_ready);
process_data();
data_ready = 0;