[英]How to pass byte slice between go routines using channels
I have a function that reads data from a source
and send them to destination
.我有一个函数可以从source
读取数据并将它们发送到destination
。 Source and destination could be anything, lets say for this example source is database (any MySQL
, PostgreSQL
...) and destination is distributed Q
(any... ActiveMQ
, Kafka
).源和目标可以是任何东西,假设在这个例子中源是数据库(任何MySQL
、 PostgreSQL
...),目标是distributed Q
(任何... ActiveMQ
、 Kafka
)。 Messages are stored in bytes.消息以字节为单位存储。
This is main function.这是主要功能。 idea is it will spin a new go routine and will wait for messages to be returned for future processing.想法是它将旋转一个新的 goroutine 并等待消息返回以供将来处理。
type Message []byte
func (p *ProcessorService) Continue(dictId int) {
level.Info(p.logger).Log("process", "message", "dictId", dictId)
retrieved := make(chan Message)
go func() {
err := p.src.Read(retrieved, strconv.Itoa(p.dictId))
if err != nil {
level.Error(p.logger).Log("process", "read", "message", "err", err)
}
}()
for r := range retrieved {
go func(message Message) {
level.Info(p.logger).Log("message", message)
if len(message) > 0 {
if err := p.dst.sendToQ(message); err != nil {
level.Error(p.logger).Log("failed", "during", "persist", "err", err)
}
} else {
level.Error(p.logger).Log("failed")
}
}(r)
}
}
and this is read function itself这是读取功能本身
func (s *Storage) Read(out chan<- Message, opt ...string) error {
// I just skip some basic database read operations here
// but idea is simple, read data from the table / file row by row and
//
for _, value := range dataFromDB {
message, err := value.row
if err == nil {
out <- message
} else {
errorf("Unable to get data %v", err)
out <- make([]byte, 0)
}
}
})
close(out)
if err != nil {
return err
}
return nil
}
As you can see communication done via out chan<- Message channel.如您所见,通过 out chan<- 消息通道完成通信。 My concern in Continue function, specifically here我对继续功能的关注,特别是这里
for r := range retrieved {
go func(message Message) {
// basically here message and r are pointing to the same underlying array
}
}
When data received var r
is a type of slice byte.当接收到数据时, var r
是一种切片字节。 Then it passed to go func(message Message)
everything passed by value in go, in this case var r
will be passed as copy to anonymous func, however it will still have a pointer to underlying slice data.然后它传递给go func(message Message)
go 中按值传递的所有内容,在这种情况下, var r
将作为副本传递给匿名 func,但它仍然有一个指向底层切片数据的指针。 I am curious if it could be a problem during p.dst.sendToQ(message);
我很好奇在p.dst.sendToQ(message);
期间是否会出现问题p.dst.sendToQ(message);
execution and at the same time read function will send something to out channel
causing slice data structure to be overridden with a new information.执行和同时读取函数将向out channel
发送一些信息,导致切片数据结构被新信息覆盖。 Should I copy byte slice r
into the new byte slice before passing to anonymous function, so underlying arrays will be different?在传递给匿名函数之前,我是否应该将字节切片r
复制到新的字节切片中,以便底层数组会有所不同? I tested it, but couldn't really cause this behavior.我测试了它,但不能真正导致这种行为。 Not sure if I am paranoid or have to worry about it.不知道我是偏执还是不得不担心。
The message
in p.dst.sendToQ(message)
is the same slice as value.row
when you get data from the db.该message
在p.dst.sendToQ(message)
是同一切片作为value.row
当你从数据库获取数据。 So, as long as each value.row
has a different underlying array, you should be good.所以,只要每个value.row
有一个不同的底层数组,你应该很好。 So, I suggest you check the source and make sure it does not use a common byte array and keeps rewriting to it.因此,我建议您检查源代码并确保它不使用公共字节数组并不断对其进行重写。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.