How to gzip and post a file without memory allocation

Question

Does anyone know how I can open a file, gzip it on the go and send it to a server using a post request?

I have a file that is a reader. Also, I have http.Post() function where I can pass the reader as a 3rd parameter. But gzip.NewWriter() expects a writer and returns a writer.

I know that I can create some intermediary buffer to connect this all but I don't want to allocate much memory because these files can be quite big. Is there a way to make this pipe and pass it to the HTTP post request?

Answer 1

Yes, io.Pipe

    reader, writer := io.Pipe()
    go func() {
        defer w.Close()

        gw := gzip.NewWriter(w)
        defer gw.Close()

        gw.Write( []byte("replace with your data"))
    }()

    http.Post("some.url", "application/zip", reader)

Answer 2

Other answers outline how to use io.Pipe. This answer shows more detail, particularly with regards to error handling.

func gzFileReader(fname string) (io.ReadCloser, error) {
    f, err := os.Open(fname)
    if err != nil {
        return nil, err
    }

    // Use io.Pipe and a goroutine to create reader
    // on data written by the appliation.
    r, w := io.Pipe()
    go func() {
        // Always close the file.
        defer f.Close()

        // Copy file through gzip to pipe writer.
        gzw := gzip.NewWriter(w)
        _, err := io.Copy(gzw, f)

        // Use CloseWithError to propgate errors back to
        // the main goroutine.
        if err != nil {
            w.CloseWithError(err)
            return
        }

        // Flush the gzip writer.
        w.CloseWithError(gzw.Close())
    }()
    return r, nil
}

Use the function like this:

body, err := gzFileReader("example.txt")
if err != nil {
    log.Fatal(err)
}
defer body.Close()

req, err := http.NewRequest("POST", "http://example.com/", body)
if err != nil {
    log.Fatal(err)
}
req.Header.Set("Content-Encoding", "gzip")
resp, err := http.DefaultClient.Do(req)

Answer 3

A net/http.Request.Body is an io.ReadCloser — hence it's any value which implements io.Reader and io.Closer interfaces. That is, you can call Read on it and then eventually Close it.

The NewWriter of compress/gzip accepts an io.Writer — an object of any type which you can Write to.

So we need to connect the "gzipped stream" writer with the HTTP request's body "reader".
That's where io.Pipe takes the stage.

So, basically:

1. Open the source file.
2. Create an io.Pipe .
3. Create a gzip writer connected to the writing end of the pipe.
4. Create an HTTP request whose body is the reading end of the pipe.
5. Call io.Copy to shovel the data between the opened file and the gzipped writer.
6. Flush the gzipped writer and close it.

Always check for errors everywere in this code.

Note that some memory buffers will inevitably be used: io.Copy uses something like 32 KiB and io.Pipe will, too.
There are ways to reduce these numbers but I'd consder this premature optimization.

Also note that since you cannot predict the side of the resulting gzipped stream, your HTTP client will use chunked transfer encoding.