堆栈内存溢出
  简体   繁体   English

Golang：http2 客户端在启用 idleConn 时立即通过代理关闭连接

[英]Golang: http2 client immediately close connections via proxy while idleConn was enabled

 原文  2021-12-05 13:49:53  6   1    go/ proxy/ squid

问题描述

0x00 TL;DR 0x00 TL;DR

I'm using Go to implement an http client and squid as a forward proxy to send requests to remote servers.我正在使用 Go 来实现 http 客户端和 squid 作为转发代理向远程服务器发送请求。 Things goes well when using http/1.1 via proxy or http/1.1, http2 without proxy, however, while using http2 client via proxy, most of the connections were closed immediately and only one or two were kept.通过代理使用 http/1.1 或 http/1.1, http2 不使用代理时一切顺利，但是，通过代理使用 http2 客户端时，大多数连接立即关闭，只保留一两个。

Not sure it's my bad code or what.不确定这是我的错误代码还是什么。 The idleConn configuration was enabled on the http transport.在 http 传输上启用了 idleConn 配置。 Thanks in advance.提前致谢。

0x01 Environments 0x01 环境

Code代码

package main

import (
    "context"
    "crypto/tls"
    "fmt"
    "io"
    "io/ioutil"
    "net"
    "net/http"
    "net/url"
    "sync"
    "time"
)

const (
    proxyURL = "http://{{proxy-ip}}:3128"
)

func main() {
    wg := &sync.WaitGroup{}

    wg.Add(1)
    go func() {
        doSendRequests()
        wg.Done()
    }()

    wg.Wait()
}

func doSendRequests() {
    //client := newHTTPClient(nil)
    client := newHTTPClient(proxy)

    round := 0
    for {
        round += 1
        for i := 0; i < 5; i++ {
            go func(r int) {
                start := time.Now()
                var err error
                defer func() {
                    duration := time.Since(start)
                    fmt.Printf("Round: %d, A: %v, duration: %v\n", r, err, duration)
                }()
                req := newRequest("https://www.google.com")
                resp, err := client.Do(req)
                if err == nil {
                    io.Copy(ioutil.Discard, resp.Body)
                    resp.Body.Close()
                } else {
                    fmt.Printf("A: %v\n", err)
                }
            }(round)
        }

        time.Sleep(100 * time.Second)
    }
}

type TLSDialer struct {
    net.Dialer
}

var (
    config = &tls.Config{InsecureSkipVerify: true}
)

func (dialer *TLSDialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error) {
    return tls.DialWithDialer(&dialer.Dialer, network, addr, config)
}

var (
    DefaultDialer = net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second,
        DualStack: true,
    }

    DefaultTLSDialer = TLSDialer{DefaultDialer}
)

func proxy(req *http.Request) (*url.URL, error) {
    uri, err := url.Parse(proxyURL)
    if err != nil {
        return nil, nil
    }
    return uri, nil
}

func newHTTPClient(proxy func(*http.Request) (*url.URL, error)) *http.Client {
    t := &http.Transport{
        DialContext:           (&DefaultDialer).DialContext,
        DisableKeepAlives:     false,
        DisableCompression:    false,
        ForceAttemptHTTP2:     true,
        MaxIdleConns:          5000,
        MaxIdleConnsPerHost:   1000,
        IdleConnTimeout:       0 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
    }

    if proxy == nil {
        t.DialTLSContext = (&DefaultTLSDialer).DialContext
    } else {
        t.Proxy = proxy
        t.TLSClientConfig = &tls.Config{
            InsecureSkipVerify: true,
        }
    }

    return &http.Client{Transport: t}
}

func newRequest(uri string) *http.Request {
    ctx, _ := context.WithTimeout(context.Background(), 60*time.Second)
    req, _ := http.NewRequestWithContext(ctx, "GET", uri, nil)

    return req
}

Env环境

go 1.15.3 go 1.15.3

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ec2-user/.cache/go-build"
GOENV="/home/ec2-user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ec2-user/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ec2-user/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build200120630=/tmp/go-build -gno-record-gcc-switches"

uname -a unname -a

Linux ip-10-53-74-64.ec2.internal 4.14.243-185.433.amzn2.x86_64 #1 SMP Mon Aug 9 05:55:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

squid 4.15鱿鱼 4.15

0x02 Phenomenon 0x02 现象

As I mentioned in the summary, everything goes well when I'm using http/1.1 via proxy or http/1.1, http2 without proxy.正如我在摘要中提到的，当我通过代理使用 http/1.1 或 http/1.1、http2 没有代理时，一切都很顺利。 However, when using http2 client via proxy, the connections will be created as many as the requests, then close others and leave only one or two connections.但是，当通过代理使用 http2 客户端时，将创建与请求一样多的连接，然后关闭其他连接，只留下一两个连接。 I capture packages with Wireshark and found that the connections were closed from the client-side initially, so it may do not matter with the proxy-side, not sure about this.我使用 Wireshark 捕获包，发现连接最初是从客户端关闭的，因此代理端可能无关紧要，对此不确定。

without proxy没有代理

// www.google.com
Every 0.5s: netstat -anop | grep '172.217'

Sat Nov 27 04:26:22 2021

tcp        0      0 10.53.74.64:44106       172.217.0.36:443        ESTABLISHED 3110/main            keepalive (25.35/0/0)
tcp        0      0 10.53.74.64:44378       172.217.0.36:443        ESTABLISHED 3110/main            keepalive (25.44/0/0)
tcp        0      0 10.53.74.64:44084       172.217.0.36:443        ESTABLISHED 3110/main            keepalive (25.60/0/0)
tcp        0      0 10.53.74.64:44374       172.217.0.36:443        ESTABLISHED 3110/main            keepalive (25.43/0/0)
tcp        0      0 10.53.74.64:44220       172.217.0.36:443        ESTABLISHED 3110/main            keepalive (25.52/0/0)

with proxy and http/1.1使用代理和 http/1.1

Every 0.5s: netstat -anop | grep ":3128"

Sun Dec  5 13:09:44 2021

tcp        0      0 10.53.74.64:58370       {{proxy-ip}}:3128     ESTABLISHED 26322/main           keepalive (23.70/0/0)
tcp        0      0 10.53.74.64:58366       {{proxy-ip}}:3128     ESTABLISHED 26322/main           keepalive (23.70/0/0)
tcp        0      0 10.53.74.64:58374       {{proxy-ip}}:3128     ESTABLISHED 26322/main           keepalive (23.70/0/0)
tcp        0      0 10.53.74.64:58368       {{proxy-ip}}:3128     ESTABLISHED 26322/main           keepalive (23.69/0/0)
tcp        0      0 10.53.74.64:58372       {{proxy-ip}}:3128     ESTABLISHED 26322/main           keepalive (23.70/0/0)

with proxy and http2使用代理和 http2

As shown below, the other connections were closed soon after ESTABLISHED and the idleConn configuration was enabled on the http transport.如下所示，其他连接在 ESTABLISHED 后不久关闭，并且在 http 传输上启用了 idleConn 配置。

Sun Dec  5 13:19:35 UTC 2021
tcp        0    284 10.53.74.64:58414       {{proxy-ip}}:3128     ESTABLISHED 34301/main           on (0.20/0/0)
tcp        0    284 10.53.74.64:58416       {{proxy-ip}}:3128     ESTABLISHED 34301/main           on (0.19/0/0)
tcp        0    284 10.53.74.64:58410       {{proxy-ip}}:3128     ESTABLISHED 34301/main           on (0.19/0/0)
tcp        0    284 10.53.74.64:58412       {{proxy-ip}}:3128     ESTABLISHED 34301/main           on (0.19/0/0)
tcp        0    284 10.53.74.64:58418       {{proxy-ip}}:3128     ESTABLISHED 34301/main           on (0.19/0/0)

Sun Dec  5 13:19:35 UTC 2021
tcp        0     89 10.53.74.64:58414       {{proxy-ip}}:3128     FIN_WAIT1   -                    on (0.14/0/0)
tcp        0     89 10.53.74.64:58416       {{proxy-ip}}:3128     FIN_WAIT1   -                    on (0.14/0/0)
tcp        0    365 10.53.74.64:58410       {{proxy-ip}}:3128     ESTABLISHED 34301/main           on (0.14/0/0)
tcp        0     89 10.53.74.64:58412       {{proxy-ip}}:3128     FIN_WAIT1   -                    on (0.14/0/0)
tcp        0     89 10.53.74.64:58418       {{proxy-ip}}:3128     FIN_WAIT1   -                    on (0.14/0/0)

Sun Dec  5 13:19:35 UTC 2021
tcp        0      0 10.53.74.64:58414       {{proxy-ip}}:3128     TIME_WAIT   -                    timewait (59.99/0/0)
tcp        0      0 10.53.74.64:58416       {{proxy-ip}}:3128     TIME_WAIT   -                    timewait (59.99/0/0)
tcp        0     31 10.53.74.64:58410       {{proxy-ip}}:3128     ESTABLISHED 34301/main           on (0.26/0/0)
tcp        0      0 10.53.74.64:58412       {{proxy-ip}}:3128     TIME_WAIT   -                    timewait (59.99/0/0)
tcp        0      0 10.53.74.64:58418       {{proxy-ip}}:3128     TIME_WAIT   -                    timewait (59.99/0/0)

This could be reproduced by the latest golang 1.17.4(by now: 2021-12-05).这可以被最新的 golang 1.17.4（现在：2021-12-05）复制。 Is there anyone who could offer a way to figure out why would this happen?有没有人可以提供一种方法来弄清楚为什么会发生这种情况？ Thanks.谢谢。

EDIT1: all the requests done without errors. EDIT1：所有请求都没有错误地完成。

EDIT2: I had open an issue on Golang repo, in case anyone interested: https://github.com/golang/go/issues/50000 EDIT2：我在 Golang repo 上打开了一个问题，以防有人感兴趣： https://github.com/golang/go/issues/50000

1 个解决方案

解决方案1
 0  2021-12-08 01:32:35

That's one familiar proxy port number.这是一个熟悉的代理端口号。

解决方案2
 0 已采纳  2021-12-12 08:20:34

Finally, I figure it out and everything is working as expected, so it's nothing to do with net/http2.最后，我弄清楚了，一切都按预期工作，所以它与 net/http2 无关。

For more details, please refer to this issue: https://github.com/golang/go/issues/50000 .更多详情请参考本期： https://github.com/golang/go/issues/50000 。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Golang http2服务器推送的高级客户端 - High level client for golang http2 server push 是否可以在 Go 中将代理/袜子与 http2 客户端一起使用？ - Is it possible to use proxy/socks with http2 client in Go? 每次请求后关闭 http2 客户端连接 - Close http2 client connection after every request 通过HTTP中间件验证WebSocket连接-Golang - Authenticating WebSocket Connections via HTTP Middleware - Golang 默认情况下，本地未启用HTTP2 - HTTP2 not enabled by default on localhost golang 正确的 http2 请求 - golang proper http2 request Golang：http2和TLS握手错误 - Golang: http2 and TLS handshake errors 无法在Golang的http2中使用流式传输 - fail to use streaming in http2 in golang 如何过早关闭/中止 Golang http.Client POST - how to close/abort a Golang http.Client POST prematurely 反向代理是否有可能将 http/1.1 连接升级到 http2？ - Is it possible that a reverse proxy upgrades a http/1.1 connection to http2?
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM