sync.Map 或使用 goroutines 时的通道

Question

I'm writing program that parses a lot of files looking for "interesting" lines. Then it's checking if these lines were seen before. Each file is parsed using separate goroutine. I'm wondering which approach is better:

1. Use sync.Map or something similar
2. Use channels and separate goroutine which should be responsible only for uniqueness check (probably using standard map). It would receive request and respond with something simple like "Not unique" or "Unique (and added)"

Is any of these solutions more popular or maybe both are wrong?

Answer 1

If you would like to have workers which can access a global map for unique checking, you can use a sync.RWMutex to be sure that the map is protected, like:

var (
  mutex sync.RWMutex = sync.RWMutex{}
  alreadySeen map[string]struct{} = make(map[string]struct{})
)

func Work() {
  for {
    Processing lines here...
    //Checking 
    mutex.RLock() //Lock for reading only
    if _, found := alreadySeen[line]; !found {
       mutex.RUnLock()
       mutex.Lock()
       alreadySeen[line] = struct{}{}
       mutex.UnLock()
    } else {
       mutex.RUnLock()
    }
  }
}

Another approach is to use a concurrent safe map to skip the whole mutexing, for example this package: https://github.com/cornelk/hashmap