[英]How do I terminate an infinite loop from inside of a goroutine?
I'm writing an app using Go that is interacting with Spotify's API and I find myself needing to use an infinite for loop to call an endpoint until the length of the returned slice is less than the limit, signalling that I've reached the end of the available entries.我正在使用 Go 编写一个应用程序,该应用程序与 Spotify 的 API 交互,我发现自己需要使用无限循环来调用端点,直到返回的切片的长度小于限制,这表明我已经到了尽头的可用条目。
For my user account, there are 1644 saved albums (I determined this by looping through without using goroutines).对于我的用户帐户,有 1644 个保存的专辑(我通过循环而不使用 goroutines 确定了这一点)。 However, when I add goroutines in, I'm getting back 2544 saved albums with duplicates.
但是,当我添加 goroutines 时,我得到了 2544 个保存的带有重复的专辑。 I'm also using the semaphore pattern to limit the number of goroutines so that I don't exceed the rate limit.
我还使用信号量模式来限制 goroutine 的数量,这样我就不会超过速率限制。
I assume that the issue is with using the active
variable rather than channels, but my attempt at that just resulted in an infinite loop我认为问题在于使用
active
变量而不是通道,但我的尝试只是导致了无限循环
wg := &sync.WaitGroup{}
sem := make(chan bool, 20)
active := true
offset := 0
for {
sem <- true
if active {
// add each new goroutine to waitgroup
wg.Add(1)
go func() error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
// release the worker
defer func() { <-sem }()
savedAlbums, err := client.CurrentUsersAlbums(ctx, spotify.Limit(50), spotify.Offset(offset))
if err != nil {
return err
}
userAlbums = append(userAlbums, savedAlbums.Albums...)
if len(savedAlbums.Albums) < 50 {
// since the limit is set to 50, we know that if the number of returned albums
// is less than 50 that we're done retrieving data
active = false
return nil
} else {
offset += 50
return nil
}
}()
} else {
wg.Wait()
break
}
}
Thanks in advance!提前致谢!
I suspect that your main issue may be a misunderstanding of what the go
keyword does;我怀疑您的主要问题可能是对
go
关键字的作用有误解; from the docs :来自文档:
A "go" statement starts the execution of a function call as an independent concurrent thread of control, or goroutine, within the same address space.
“go”语句开始执行 function 调用作为独立并发控制线程或 goroutine,在同一地址空间内。
So go func() error {
starts the execution of the closure;所以
go func() error {
开始执行闭包; it does not mean that any of the code runs immediately.这并不意味着任何代码都会立即运行。 In fact because,
client.CurrentUsersAlbums
will take a while, it's likely you will be requesting the first 50 items 20 times.事实上,因为
client.CurrentUsersAlbums
需要一段时间,所以您可能会请求前 50 个项目 20 次。 This can be demonstrated with a simplified version of your application ( playground )这可以通过您的应用程序的简化版本(操场)来演示
func main() {
wg := &sync.WaitGroup{}
sem := make(chan bool, 20)
active := true
offset := 0
for {
sem <- true
if active {
// add each new goroutine to waitgroup
wg.Add(1)
go func() error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
// release the worker
defer func() { <-sem }()
fmt.Println("Getting from:", offset)
time.Sleep(time.Millisecond) // Simulate the query
// Pretend that we got back 50 albums
offset += 50
if offset > 2000 {
active = false
}
return nil
}()
} else {
wg.Wait()
break
}
}
}
Running this will produce somewhat unpredictable results (note that the playground caches results so try it on your machine) but you will probably see 20 X Getting from: 0
.运行它会产生一些不可预知的结果(请注意,playground 会缓存结果,因此请在您的机器上尝试),但您可能会看到 20 X
Getting from: 0
。
A further issue is data races .另一个问题是数据竞争。 Updating a variable from multiple goroutines without protection (eg
sync.Mutex
) results in undefined behaviour.在没有保护的情况下从多个 goroutine 更新变量(例如
sync.Mutex
)会导致未定义的行为。
You will want to know how to fix this but unfortunately you will need to rethink your algorithm.你会想知道如何解决这个问题,但不幸的是你需要重新考虑你的算法。 Currently the process you are following is:
目前您正在遵循的过程是:
pos
to 0pos
设置为 050
records starting from pos
pos
开始获取50
条记录50
records then pos=pos+50
and loop back to step 2
50
条记录,则pos=pos+50
并loop back to step 2
This is a sequential algorithm;这是一个顺序算法; you don't know whether you have all of the data until you have requested the previous section.
在您请求上一部分之前,您不知道您是否拥有所有数据。 I guess you could make speculative queries (and handle failures) but a better solution would be to find some way to determine the number of results expected and then split the queries to get that number of records between multiple goroutines.
我想您可以进行推测性查询(并处理故障),但更好的解决方案是找到某种方法来确定预期结果的数量,然后拆分查询以在多个 goroutine 之间获取该数量的记录。
go func(offset int) error {
.go func(offset int) error {
.chan bool
chan bool
this is the example: https://go.dev/play/p/RhZMDYmsVD3这是一个例子: https://go.dev/play/p/RhZMDYmsVD3
if applied to your code:如果应用于您的代码:
wg := &sync.WaitGroup{}
worker := 20
active := make(chan bool, worker)
for i := 0; i < worker; i++ {
active <- true
}
offset := 0
for {
if <-active {
// add each new goroutine to waitgroup
wg.Add(1)
go func(offset int) error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
savedAlbums, err := client.CurrentUsersAlbums(ctx, spotify.Limit(50), spotify.Offset(offset))
if err != nil {
// active <- false // maybe you need this
return err
}
userAlbums = append(userAlbums, savedAlbums.Albums...)
if len(savedAlbums.Albums) < 50 {
// since the limit is set to 50, we know that if the number of returned albums
// is less than 50 that we're done retrieving data
active <- false
return nil
} else {
active <- true
return nil
}
}(offset)
offset += 50
} else {
wg.Wait()
break
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.