Go 当 goroutine 数量增加时程序变慢

Question

I'm doing a small project for my parallelism course and I have tried it with buffered channels, unbuffered channels, without channels using pointers to slices etc. Also, tried to optimize it as much as possible (not the current state) but I still get the same result: increasing number of goroutines (even by 1) slows down the whole program. Can someone please tell me what I'm doing wrong and is even parallelism enhancement possible in this situation?

Here is part of the code:

func main() {

    rand.Seed(time.Now().UnixMicro())

    numAgents := 2

    fmt.Println("Please pick a number of goroutines: ")
    fmt.Scanf("%d", &numAgents)

    numFiles := 4
    fmt.Println("How many files do you want?")
    fmt.Scanf("%d", &numFiles)
    start := time.Now()

    numAssist := numFiles
    channel := make(chan []File, numAgents)
    files := make([]File, 0)

    for i := 0; i < numAgents; i++ {
        if i == numAgents-1 {
            go generateFiles(numAssist, channel)
        } else {
            go generateFiles(numFiles/numAgents, channel)
            numAssist -= numFiles / numAgents
        }
    }

    for i := 0; i < numAgents; i++ {
        files = append(files, <-channel...)
    }

    elapsed := time.Since(start)
    fmt.Printf("Function took %s\n", elapsed)
}

func generateFiles(numFiles int, channel chan []File) {
    magicNumbersMap := getMap()
    files := make([]File, 0)

    for i := 0; i < numFiles; i++ {
        content := randElementFromMap(&magicNumbersMap)

        length := rand.Intn(400) + 100
        hexSlice := getHex()

        for j := 0; j < length; j++ {
            content = content + hexSlice[rand.Intn(len(hexSlice))]
        }

        hash := getSHA1Hash([]byte(content))

        file := File{
            content: content,
            hash:    hash,
        }

        files = append(files, file)
    }

    channel <- files

}

Expectation was that by increasing goroutines the program would run faster but to a certain number of goroutines and at that point by increasing goroutines I would get the same execution time or a little bit slower.

EDIT: All the functions that are used:

    import (
    "crypto/sha1"
    "encoding/base64"
    "fmt"
    "math/rand"
    "time"
)

type File struct {
    content string
    hash    string
}

func getMap() map[string]string {
    return map[string]string{
        "D4C3B2A1": "Libcap file format",
        "EDABEEDB": "RedHat Package Manager (RPM) package",
        "4C5A4950": "lzip compressed file",
    }
}

func getHex() []string {
    return []string{
        "0", "1", "2", "3", "4", "5",
        "6", "7", "8", "9", "A", "B",
        "C", "D", "E", "F",
    }
}

func randElementFromMap(m *map[string]string) string {
    x := rand.Intn(len(*m))
    for k := range *m {
        if x == 0 {
            return k
        }
        x--
    }
    return "Error"
}

func getSHA1Hash(content []byte) string {
    h := sha1.New()
    h.Write(content)
    return base64.URLEncoding.EncodeToString(h.Sum(nil))
}

Answer 1

Simply speaking - the files generation code is not complex enough to justify parallel execution. All the context switching and moving data through the channel eats all benefit of parallel processing.

If you add something like time.Sleep(time.Millisecond * 10) inside the loop in your generateFiles function as if it was doing something more complex, you'll see what you expected to see - more goroutines work faster. But again, only until certain level, when extra work to do parallel processing overweights the benefit.

Note also, the execution time of the last bit of your program:

for i := 0; i < numAgents; i++ {
    files = append(files, <-channel...)
}

directly depends on number of goroutines. Since all goroutines finish approximately at the same time, this loop almost never executed in parallel with your workers and the time it takes to run is simply added to the total time.

Next, when you append to files slice multiple times, it has to grow several times and copy the data over to the new location. You can avoid this by initially creating a slice that will fil all your resulting elements (luckily, you know exactly how many you'll need).