如何查看与 HTTP 客户端的 HTTP/HTTPS 交互

Question

How do you view what exactly is sent over the wire by a HTTP client and how the connection is configured?

Answer 1

go has packages called httputil & httptrace that can facilitate viewing the HTTP lifecycle, as well as what is actually sent over the wire: http-tracing blog post httptrace go doc httputil go doc

NOTE httputil.DumpRequestOut is meant for outgoing messages on the client side and httputil.DumpRequest are meant for incoming messages on the server side

NOTE httputil.DumpRequestOut appends the default transport's headers, so if you customize the transport, the changes would not be reflected. See: Why does the HTTP Client Force an Accept-Encoding header

Sample Implementation:

package main

import (
    "crypto/tls"
    "fmt"
    "net/http"
    "net/http/httptrace"
    "net/http/httputil"
    "net/textproto"
    "time"
)

func main() {
    url := "https://www.google.com"
    client := &http.Client{}
    req, err := http.NewRequest(http.MethodGet, url, nil)
    if err != nil {
        return
    }
    requestDump, err := httputil.DumpRequestOut(req, false)
    if err != nil {
        fmt.Printf("%s: REQUEST ERR: %s\n", time.Now(), err)
    }
    fmt.Printf("%s: REQUEST: \n%s\n", time.Now(), string(requestDump))
    trace := &httptrace.ClientTrace{
        // GetConn is called before a connection is created or
        // retrieved from an idle pool. The hostPort is the
        // "host:port" of the target or proxy. GetConn is called even
        // if there's already an idle cached connection available.
        GetConn: func(hostPort string) {
            fmt.Printf("Get Conn: hostPort: %s\n", hostPort)
        },
        // GotConn is called after a successful connection is
        // obtained. There is no hook for failure to obtain a
        // connection; instead, use the error from
        // Transport.RoundTrip.
        GotConn: func(connInfo httptrace.GotConnInfo) {
            fmt.Printf("Got Conn: connInfo: %+v\n", connInfo)
        },
        // PutIdleConn is called when the connection is returned to
        // the idle pool. If err is nil, the connection was
        // successfully returned to the idle pool. If err is non-nil,
        // it describes why not. PutIdleConn is not called if
        // connection reuse is disabled via Transport.DisableKeepAlives.
        // PutIdleConn is called before the caller's Response.Body.Close
        // call returns.
        // For HTTP/2, this hook is not currently used.
        PutIdleConn: func(err error) {
            fmt.Printf("PutIdlConn: ERR: %s\n", err)
        },
        // GotFirstResponseByte is called when the first byte of the response
        // headers is available.
        GotFirstResponseByte: func() {
            fmt.Println("GotFirstResponseByte")
        },
        // Got100Continue is called if the server replies with a "100
        // Continue" response.
        Got100Continue: func() {
            fmt.Println("Got100Continue")
        },
        // Got1xxResponse is called for each 1xx informational response header
        // returned before the final non-1xx response. Got1xxResponse is called
        // for "100 Continue" responses, even if Got100Continue is also defined.
        // If it returns an error, the client request is aborted with that error value.
        Got1xxResponse: func(code int, header textproto.MIMEHeader) error {
            fmt.Printf("Got1xxResponse: code: %d header: %+v\n", code, header)
            return nil
        },
        // DNSStart is called when a DNS lookup begins.
        DNSStart: func(dnsInfo httptrace.DNSStartInfo) {
            fmt.Printf("DNS Start: dnsInfo: %+v\n", dnsInfo)
        },
        // DNSDone is called when a DNS lookup ends.
        DNSDone: func(dnsInfo httptrace.DNSDoneInfo) {
            fmt.Printf("DNS Done: dnsInfo: %+v\n", dnsInfo)
        },
        // ConnectStart is called when a new connection's Dial begins.
        // If net.Dialer.DualStack (IPv6 "Happy Eyeballs") support is
        // enabled, this may be called multiple times.
        ConnectStart: func(network, addr string) {
            fmt.Printf("Connect Start: Network Addr: %s %s\n", network, addr)
        },
        // ConnectDone is called when a new connection's Dial
        // completes. The provided err indicates whether the
        // connection completedly successfully.
        // If net.Dialer.DualStack ("Happy Eyeballs") support is
        // enabled, this may be called multiple times.
        ConnectDone: func(network, addr string, err error) {
            fmt.Printf("Connect Done: Network Addr: %s %s ERR: %s\n", network, addr, err)
        },
        // TLSHandshakeStart is called when the TLS handshake is started. When
        // connecting to an HTTPS site via an HTTP proxy, the handshake happens
        // after the CONNECT request is processed by the proxy.
        TLSHandshakeStart: func() {
            fmt.Println("TLSHandshakeStart")
        },
        // TLSHandshakeDone is called after the TLS handshake with either the
        // successful handshake's connection state, or a non-nil error on handshake
        // failure.
        TLSHandshakeDone: func(connState tls.ConnectionState, err error) {
            fmt.Printf("TLSHandshakeDone: connState: %+v ERR: %s\n", connState, err)
        },
        // WroteHeaderField is called after the Transport has written
        // each request header. At the time of this call the values
        // might be buffered and not yet written to the network.
        WroteHeaderField: func(key string, value []string) {
            fmt.Printf("WroteHeaderField: key: %s val: %s\n", key, value)
        },
        // WroteHeaders is called after the Transport has written
        // all request headers.
        WroteHeaders: func() {
            fmt.Println("WroteHeaders")
        },
        // Wait100Continue is called if the Request specified
        // "Expect: 100-continue" and the Transport has written the
        // request headers but is waiting for "100 Continue" from the
        // server before writing the request body.
        Wait100Continue: func() {
            fmt.Println("Wait100Continue")
        },
        // WroteRequest is called with the result of writing the
        // request and any body. It may be called multiple times
        // in the case of retried requests.
        WroteRequest: func(info httptrace.WroteRequestInfo) {
            fmt.Printf("WroteRequest: %+v\n", info)
        },
    }
    req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))

    resp, err := client.Do(req)
    fmt.Printf("%s: RESPONSE OBJ: \n%v\n", time.Now(), resp)
}

Output:

2020-07-29 14:09:53.682167 -0700 PDT m=+0.000769969: REQUEST:
GET / HTTP/1.1
Host: www.google.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


Get Conn: hostPort: www.google.com:443
DNS Start: dnsInfo: {Host:www.google.com}
DNS Done: dnsInfo: {Addrs:[{IP:172.217.17.100 Zone:} {IP:2a00:1450:400e:806::2004 Zone:}] Err:<nil> Coalesced:false}
Connect Start: Network Addr: tcp 172.217.17.100:443
Connect Done: Network Addr: tcp 172.217.17.100:443 ERR: %!s(<nil>)
TLSHandshakeStart
TLSHandshakeDone: connState: {Version:772 HandshakeComplete:true DidResume:false CipherSuite:4865 NegotiatedProtocol:h2 NegotiatedProtocolIsMutual:true ServerName: PeerCertificates:[0xc0001d6000 0xc0001d6580] VerifiedChains:[[0xc0001d6000 0xc0001d6580 0xc000278b00]] SignedCertificateTimestamps:[] OCSPResponse:[] ekm:0x1226ae0 TLSUnique:[]} ERR: %!s(<nil>)
Got Conn: connInfo: {Conn:0xc0001a2000 Reused:false WasIdle:false IdleTime:0s}
WroteHeaderField: key: :authority val: [www.google.com]
WroteHeaderField: key: :method val: [GET]
WroteHeaderField: key: :path val: [/]
WroteHeaderField: key: :scheme val: [https]
WroteHeaderField: key: accept-encoding val: [gzip]
WroteHeaderField: key: user-agent val: [Go-http-client/2.0]
WroteHeaders
WroteRequest: {Err:<nil>}
GotFirstResponseByte
2020-07-29 14:09:54.620195 -0700 PDT m=+0.938796345: RESPONSE OBJ:
&{200 OK 200 HTTP/2.0 2 0 map[Alt-Svc:[h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private, max-age=0] Content-Type:[text/html; charset=ISO-8859-1] Date:[Wed, 29 Jul 2020 21:09:54 GMT] Expires:[-1] P3p:[CP="This is not a P3P policy! See g.co/p3phelp for more info."] Server:[gws] Set-Cookie:[1P_JAR=2020-07-29-21; expires=Fri, 28-Aug-2020 21:09:54 GMT; path=/; domain=.google.com; Secure NID=204=qnJT-6IGam7-C1fTR8uIkbDPnfV7OwgOGn5-6tGCWLYmeaRMoSKgV1qSRfKGLghNgQVWY9N_o6hUWKm69I5KrdVqIEVVxRy6XSY6F4c1JyTJZZqEMxMlkpznu-PWOn9eAezKBONTxCZgsGZYboEeYZ5-qZBjUvd7BratNIPkTxU; expires=Thu, 28-Jan-2021 21:09:54 GMT; path=/; domain=.google.com; HttpOnly] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] 0xc00018c1e0 -1 [] false true map[] 0xc000112100 0xc00007c000}