如何使用 goroutine 和 channel 批量读取 mysql 数据

Question

I'm a newbee for golang, now need to read a big amount data in mysql, so I wanna use goroutine and channel to get data in high performance, but don't know how to avoid data duplication for each goroutine and make whole process stable. for instance, table schema is as below, I wanna get all records which create_time is smaller than 1000000000000000000 , I wanna create 10 goroutines and read data concurrently, each goroutine do some business logic, how to design codes? thank u

id content last_id create_time

Answer 1

I would suggest you to create a goroutine that would publish the data to your channel. Then you can add listener go routines to handle the published data. This can be done as following:

Main:

const GoroutineCount = 10

type SomeData []int

func main() {
    ch := make(chan SomeData, 1)

    go PublishData(ch)

    for i := 0; i < GoroutineCount; i++ {
        go ProcessData(ch)
    }
}

For assumptions, I have used a simple slice of int as data. This can be slice of any struct as required.

Publish data to channel:

const ChunkSize = 1000

func PublishData(ch chan SomeData) {
    // Assume having 10000 records in result set
    // This has to come from db transaction
    res := make([]int, 10000)

    // split into chunks of 1000
    chunks := GetChunk(res)

    // write chunk data to channel
    for i := range chunks {
        ch <- chunks[i]
    }
}

func GetChunk(input SomeData) []SomeData {
    var result []SomeData

    boundary := len(input)
    index := 0
    for index = 0; boundary >= ChunkSize; index+=ChunkSize {
        boundary -= ChunkSize
        lastIndex := index+ChunkSize
        result = append(result, input[index:lastIndex])
    }
    boundary = len(input) % ChunkSize
    if boundary > 0 {
        lastIndex := index+ boundary
        result = append(result, input[index:lastIndex])
    }

    return result
}

Process individual chunks as:

func ProcessData(ch chan SomeData) {
    // Read single chunk
    res := <-ch

    // Process chunk data
    fmt.Printf("len %d\n", len(res))
}

Code on go playground: https://play.golang.org/p/X9Q991h6ru_n