Golang - Have some troubles with Go-routines and channels

Question

I'm kinda new to Golang and trying to develop a program that uploads images async to imgur. However I'm having some difficulties with my code.

So this is my task;

func uploadT(url string,c chan string, d chan string)  {

    var subtask string
    subtask=upload(url)

    var status string
    var url string

    if subtask!=""{
        status = "Success!"
        url =subtask

    } else {
        status = "Failed!"
        url =subtask
    }

    c<-url
    d<-status
}

And here is my POST request loop for async uploading;

c:=make(chan string, len(js.Urls))
d:=make(chan string, len(js.Urls))

wg:=sync.WaitGroup{}
for i := range js.Urls{
    wg.Add(1)
    go uploadTask(js.Urls[i],c,d)
    //Below commented out code is slowing down the routine therefore, I commented out.
    //Needs to be working as well, however, it can work if I put this on task as well. I think I'm kinda confused with this one as well
    //pol=append(pol,retro{Url:<-c,Status:<-d})
}
<-c
<-d
wg.Done()
FinishedTime := time.Now().UTC().Format(time.RFC3339)
qwe=append(qwe,outputURLs{
               jobID:jobID,
               retro:pol,
               CreateTime: CreateTime,
               FinishedTime: FinishedTime,
           })
fmt.Println(jobID)

So I think my channels and routine does not work. It does print out jobID before the upload tasks. And also uploads seems too slow for async uploading.

I know the code is kinda mess, sorry for that. Any help is highly appreciated! Thanks in advance!

Answer 1

You're actually not using WaitGroup correctly. Everytime you call wg.Done() its actually subtracting 1 from the previous wg.Add to determine that a given task is complete. Finally, you'll need a wg.Wait() to synchronously wait for all tasks. WaitGroups are typically for fan out usage of running multiple tasks in parallel.

The simplest way based on your code example is to pass in the wg into your task, uploadT and call wg.Done() inside of the task. Note that you'll also want to use a pointer instead of the struct value.

The next implementation detail is to call wg.Wait() outside of the loop because you want to block until all the tasks are complete since all your tasks are ran with go which makes it async. If you don't wg.Wait() , it will log the jobID immediately like you said. Let me know if that is clear.

As a boilerplate, it should look something like this

func task(wg *sync.WaitGroup) {
    wg.Done()
}

wg := &sync.WaitGroup{}
for i := 0; i < 10; i++ {
    wg.Add(1)
    go task(wg)
}

wg.Wait()
// do something after the task is done
fmt.Println("done")

The other thing I want to note is that in your current code example, you're using channels but you're not doing anything with the values you're pushing into the channels so you can technically remove them.

Answer 2

Your code is kinda confusing. But if I understand correctly what you are trying to do, you are processing a list of requests and want to return the url and status of each request and time each request completed. And you want to process these in parallel.

You don't need to use WaitGroups at all. WaitGroups are good when you just want to run a bunch of tasks without bothering with the results, just want to know when everything is done. But if you are returning results, channels are sufficient.

Here is an example code that does what I think you are trying to do

package main

import (
    "time"
    "fmt"
)

type Result struct {
    URL      string
    Status   string
    Finished string
}

func task(url string, c chan string, d chan string) {
    time.Sleep(time.Second)
    c <- url
    d <- "Success"
}

func main() {
    var results []Result
    urls := []string{"url1", "url2", "url3", "url4", "url5"}
    c := make(chan string, len(urls))
    d := make(chan string, len(urls))
    for _, u := range urls {
        go task(u, c, d)
    }
    for i := 0; i < len(urls); i++ {
        res := Result{}
        res.URL = <-c
        res.Status = <-d
        res.Finished = time.Now().UTC().Format(time.RFC3339)
        results = append(results, res)
    }
    fmt.Println(results)
}

You can try it in the playground https://play.golang.org/p/N3oeA7MyZ8L

That said, this is a bit fragile. You are making channels the same size as your url list. This would work fine for a few urls, but if you have a list of a million urls you will make a rather large channel. You might want to fix the channel buffer size to some reasonable value and check whether or not channel is ready for processing before sending your request. This way you would avoid making a million requests all at once.