正确使用 fasthttp.Client 结合 goroutines

Question

I am new to Go and am looking for the correct way of using net/http or fasthttp with goroutines. Unfortunately there are not many fasthttp client examples out there.

I found the following code: ( Example1 )

package main

import (
    "bufio"
    "fmt"
    "github.com/valyala/fasthttp"
    "log"
    "net"
    "os"
    "sync"
    "time"
)

func grabPage(fastClient *fasthttp.Client, i int, wg *sync.WaitGroup) {
    defer wg.Done()
    _, body, err := fastClient.GetTimeout(nil, "https://en.wikipedia.org/wiki/Immanuel_Kant", time.Duration(time.Second*20))
    if err != nil {
        log.Fatal(err)
    }
    f, err := os.Create(fmt.Sprintf("./data/%d.txt", i))
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()
    w := bufio.NewWriter(f)
    w.Write(body)
}

func main() {
    var wg sync.WaitGroup
    total := 500

    c := &fasthttp.Client{
        Dial: func(addr string) (net.Conn, error) {
            return fasthttp.DialTimeout(addr, time.Second*10)
        },
        MaxConnsPerHost: total,
    }

    wg.Add(total)
    for index := 0; index < total; index++ {
        go grabPage(c, index, &wg)
    }
    wg.Wait()
}

In this code the developer creates a fasthttp.Client instance in the main() function and passes it to the goroutine using go grabPage(c, ...) . For my understanding, this way you create one instance and all the requests use this one instance to do the job.

On another page, a developer uses something like that: ( Example2 )

func grabPage(i int, wg *sync.WaitGroup) {
    defer wg.Done()

    fastClient := &fasthttp.Client{
        Dial: func(addr string) (net.Conn, error) {
            return fasthttp.DialTimeout(addr, time.Second*10)
        },
        MaxConnsPerHost: 500,
    }

    _, body, err := fastClient.GetTimeout(nil, "https://en.wikipedia.org/wiki/Immanuel_Kant", time.Duration(time.Second*20))
    if err != nil {
        log.Fatal(err)
    }
    f, err := os.Create(fmt.Sprintf("./data/%d.txt", i))
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()
    w := bufio.NewWriter(f)
    w.Write(body)
}

The big question is, are both solutions correct? Or does the Example2 solution really create a new instance and use a lot of memory for every goroutine?

I made the examples out of snippets for my question, in the Example2 for sure defer are missing. This is not part of the question.

A small side question: (fastClient *fasthttp.Client, i int, wg *sync.WaitGroup) -> fastClient and wg are pointers, so why call grabPage(c, index, &wg) and not grabPage(&c, index, &wg) ?

Answer 1

The big answer: both are correct (as in they work just fine), just different.

Per the docs , a fasthttp.Client is safe for concurrent use so sharing one instance is fine. It may run into concurrent connection limits but that might not be an issue.

The second example does have some overhead and will not be able to reuse connections or parameters, but again this could be a use case where it does not matter (if I only perform two operations, saving on the overhead might not be worth optimizing for).

For the second part of the question:

• c is already a *fasthttp.Client , so there's not need to take its address ( &fasthttp.Client returns a pointer to a new fasthttp.Client )
• wg is a plain sync.WaitGroup so the address must be taken