Distribute the same keyword to multiple goroutines

Question

I have something like this mock (code below) which distributes the same keyword out to multiple goroutines, except the goroutines all take different amount of times doing things with the keyword but can operate independently of each other so they don't need any synchronization. The solution given below to distribute clearly synchronizes the goroutines.

I just want to toss this idea out there to see how other people would deal with this type of distribution, as I assume it is fairly common and someone else has thought about it before.

Here are some other solutions I have thought up and why they seem kinda meh to me:

One goroutine for each keyword

Each time a new keyword comes in spawn a goroutine to handle the distribution

Give the keyword a bitmask or something for each goroutine to update

This way once all of the workers have touched the keyword it can be deleted and we can move on

Give each worker its own stack to work off of

This seems kinda appealing, just give each worker a stack to add each keyword to, but we would eventually run into a problem of a ton of memory being taken up since it is planned to run so long

The problem with all of these is that my code is supposed to run for a long time, unwatched, and that would lead to either a huge build up of keywords or goroutines due to the lazy worker taking longer than the others. It almost seems like it'd be nice to give each worker its own Amazon SQS queue or implement something similar to that myself.

EDIT:

Store the keyword outside the program

I just thought of doing it this way instead, I could perhaps just store the keyword outside the program until they all grab it and then delete it once it has been used up. This sits ok with me actually, I don't have a problem with using up disk space

Anyway here is an example of the approach that waits for all to finish:

package main

import (
    "flag"
    "fmt"
    "math/rand"
    "os"
    "os/signal"
    "strconv"
    "time"
)

var (
    shutdown chan struct{}
    count    = flag.Int("count", 5, "number to run")
)

type sleepingWorker struct {
    name  string
    sleep time.Duration
    ch    chan int
}

func NewQuicky(n string) sleepingWorker {
    var rq sleepingWorker
    rq.name = n
    rq.ch = make(chan int)
    rq.sleep = time.Duration(rand.Intn(5)) * time.Second
    return rq
}

func (r sleepingWorker) Work() {
    for {
        fmt.Println(r.name, "is about to sleep, number:", <-r.ch)
        time.Sleep(r.sleep)
    }
}

func NewLazy() sleepingWorker {
    var rq sleepingWorker
    rq.name = "Lazy slow worker"
    rq.ch = make(chan int)
    rq.sleep = 20 * time.Second
    return rq
}

func distribute(gen chan int, workers ...sleepingWorker) {
    for kw := range gen {
        for _, w := range workers {
            fmt.Println("sending keyword to:", w.name)
            select {
            case <-shutdown:
                return
            case w.ch <- kw:
                fmt.Println("keyword sent to:", w.name)
            }
        }
    }
}

func main() {
    flag.Parse()
    shutdown = make(chan struct{})
    go func() {
        c := make(chan os.Signal, 1)
        signal.Notify(c, os.Interrupt)
        <-c
        close(shutdown)
    }()

    x := make([]sleepingWorker, *count)
    for i := 0; i < (*count)-1; i++ {
        x[i] = NewQuicky(strconv.Itoa(i))
        go x[i].Work()
    }
    x[(*count)-1] = NewLazy()
    go x[(*count)-1].Work()

    gen := make(chan int)
    go distribute(gen, x...)
    go func() {
        i := 0
        for {
            i++
            select {
            case <-shutdown:
                return
            case gen <- i:
            }
        }
    }()
    <-shutdown
    os.Exit(0)
}

Answer 1

Let's assume I understand the problem correctly:

There's not too much you can do about it I'm afraid. You have limited resources (assuming all resources are limited) so if data to your input is written faster then you process it, there will be some synchronisation needed. At the end the whole process will run as quickly as the slowest worker anyway.

If you really need data from the workers available as soon as possible, the best you can do is to add some kind of buffering. But the buffer must be limited in size (even if you run in the cloud it would be limited by your wallet) so assuming never ending torrent of input it will only postpone the choke until some time in the future where you will start seeing "synchronisation" again.

All the ideas you presented in your questions are based on buffering the data. Even if you run a routine for every keyword-worker pair, this will buffer one element in every routine and, unless you implement the limit on total number of routines, you'll run out of memory. And even if you always leave some room for the quickest worker to spawn a new routine, the input queue won't be able to deliver new items as it would be choked on the slowest worker.

Buffering would solve your problem if on average you input is slower than processing time, but you have occasional spikes. If your buffer is big enough you can than accommodate the increase of throughput and maybe your quickest worker won't notice a thing.

Solution?

As go comes with buffered channels, this is the easiest to implement (also suggested by icza in the comment). Just give each worker a buffer. If you know which worker is the slowest, you can give it a bigger buffer. In this scenario you're limited by the memory of your machine.

If you're not happy with the single-machine memory limit then yes, per one of your ideas, you can "simply" store the buffer (queue) for each worker on the hard drive. But this is also limited and just postpones the blocking scenario until later. This is essentially the same as your Amazon SQS proposal (you could keep buffer in the cloud, but you need either limit it reasonably or prepare for the bill.)

The final note, depending on the system you're building, it might be not a good idea to buffer items in such a massive scale allowing to build up the backlog for the slower workers – it's often not desirable to have a worker hours, days, weeks behind the input flow and this is what would happen with an infinite buffer. The real answer then would be: improve your slowest worker to process things faster. (And add some buffering to improve the experience.)