Request progress in http headers in go

Question

It is possible for the server to send HTTP headers multiple times in response before starting to send the body. For example clickhouse does this to report progress.

From https://clickhouse.tech/docs/en/interfaces/http/amp/ :

X-ClickHouse-Progress: {"read_rows":"2752512","read_bytes":"240570816","total_rows_to_read":"8880128"}
X-ClickHouse-Progress: {"read_rows":"5439488","read_bytes":"482285394","total_rows_to_read":"8880128"}
X-ClickHouse-Progress: {"read_rows":"8783786","read_bytes":"819092887","total_rows_to_read":"8880128"}

How do I read updating headers in Go? When I tried to get the values of response.Header multiple times it does stays the same.

Here is the code fragment I use to try to get updating headers.

    //
    // Create query
    //

    progressQ := fmt.Sprintf("SELECT * from system.numbers limit 10000000")
    ctx, cancel := context.WithTimeout(be.ctx, time.Second*15)
    defer cancel()

    //
    // Prepare HTTP request
    //

    req, err := http.NewRequestWithContext(ctx, http.MethodPost, be.url, strings.NewReader(progressQ))
    if err != nil {
        return err
    }

    //
    // Set URL query
    //

    q := req.URL.Query()

    q.Set("wait_end_of_query", "1")
    q.Set("send_progress_in_http_headers", "1")

    req.URL.RawQuery = q.Encode()

    //
    // Perform request
    //
    resp, err := be.cli.Do(req)
    if err != nil {
        return err
    }
    for {
        select {
        case <-time.After(time.Second / 10):
            fmt.Println(resp.Header.Values("X-Clickhouse-Summary"))
        case <-ctx.Done():
            goto after
        }
    }
after:

The result looks like

[{"read_rows":"10002432","read_bytes":"80019456","written_rows":"0","written_bytes":"0","total_rows_to_read":"0"}]
[{"read_rows":"10002432","read_bytes":"80019456","written_rows":"0","written_bytes":"0","total_rows_to_read":"0"}]
[{"read_rows":"10002432","read_bytes":"80019456","written_rows":"0","written_bytes":"0","total_rows_to_read":"0"}]

Which differs from the example I have seen in https://altinity.com/blog/2018/9/28/progress-reports-for-long-running-queries-via-http-protocol

Answer 1

The standard library cannot do what you're asking.

The standard library's HTTP client waits until all headers are read, before returning the *http.Response object.

You'll need to implement your own HTTP client to get this behavior.

You may be able to take advantage of the DialContext field of the http.Transport type to wrap the standard network layer used by the standard http.Client . This could allow you to inspect the response before it's parsed by the standard library, potentially reporting headers as they are read.

Answer 2

I see the problem with your snippet: you're requesting "X-Clickhouse-Summary" when progress is reported in "X-ClickHouse-Progress" . If you change the header name, it should return you the slice of header values you're expecting to get.

As @Flimzy mentioned, golang HTTP standard library will return all headers at once. So there's no extra point in the loop.