How to replace goroutines loop by channels in Go

Question

I have a loop, for each iteration I have a new channel source, that I should handle. Ok, it's better to show my code. I have list of files, each file I want to tail (like tail -f). I am using github.com/ActiveState/tail package.

for _, tailFile := range files {
    t, _ := tail.TailFile(tailFile, c)

    // Goroutine per tailing file
    go func() {
        for line := range t.Lines { // t.Lines is a channel
            // Do some magic here
        }
    }()
}

I can have thousands of files and I want to run my tail in parallel. As you see my program will have thousands of goroutines. Can this loop be changed to channels, with only 1 goroutine?

Answer 1

You will find a similar approach (one goroutine per file) in the blog post pipelines .

The MD5All implementation in parallel.go starts a new goroutine for each file. In a directory with many large files, this may allocate more memory than is available on the machine.

We can limit these allocations by bounding the number of files read in parallel. In bounded.go , we do this by creating a fixed number of goroutines for reading files.
Our pipeline now has three stages: walk the tree, read and digest the files, and collect the digests.

You could organize your own code to use a limited number of goroutines, if you find yourself limited by memory allocated by a too large number of goroutines. (the goroutine itself is cheap, but the memory allocated for the "magic" part can be big)