How to let user download file while it's being generated

Question

I want to let the user start downloading a file when it's not ready yet. I don't want to send user to some page saying "Wait 30 seconds, file is being prepared." I can't generate the file in advance. I need user to click/send form, choose download location and start downloading. Generated file will be zip, so I imagine, that it should be possible to send file name with first few bytes of zips (which are always same) and before the file is generated don't confirm that tcp packet was send correctly or something like that, and after the file is generated send the rest.

How do I do that? Is there any tool which can do that. Or is there some better way? More high level solution better, C isn't my strong suit. Preferably in Python. Thanks.

File being generated is zip and before it's prepared there isn't really anything to send yet. Basically according to input I generate set of files (which takes few dozens seconds), than I zip them and serve to user. My application is in python on linux but which server I'll use isn't really important.

Answer 1

The client would most likely timeout(or wait 30 seconds without notification) while the file was prepared. I would use a stream compression algorithm(gzip) to compress the file(s) while in transit. This would not result in the best compression, but would serve the files in a predictable manner.

Investigate the "Content-Encoding: gzip" HTTP header.

Answer 2

Usually, application like this will be implement in two parts, like a ticket system.

1. When user click/submit the form, the form will send request to a service that will start generating the file as a background process, then (without waiting for the file to generate) it will generate a ticket/hash that represent the new file and then redirect user to a new URL, eg /files/<random-hash>

2. On this new URL, /files/<random-hash> , while the files is not ready, it'll return a simple HTML page that show message to user to wait, and have a script on the page that will keep refreshing the page every few seconds. As long as the files is not ready, it'll keep showing this message, but once the file is ready, this URL will return the actual files content in its response with appropriate mime-header instead.

The solution is quite simple to implement using a database and some programming. Though, if you're looking for an already-made tool ready to use, I'm sorry I'm not familiar with one. Hope this help.

Answer 3

Despite claims it's impossible, I managed to find a way. I learned a bit of Go in a meantime, so I used that, but I guess it won't be too different in other languages.

Basically first byte is written in the writer and then flushed. Browser than waits for rest.

package main

import (
    "bytes"
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "strings"
    "time"
)

func Zip(w http.ResponseWriter, r *http.Request) {
    file_name := r.URL.Path
    file_name = strings.TrimPrefix(file_name, "/files/")

    w.Header().Set("Content-type", "application/zip")
    w.Write([]byte{80})
    if f, ok := w.(http.Flusher); ok {
        f.Flush()
    }
    for {
        if _, err := os.Stat("./files/" + file_name); err == nil {
            fmt.Println("file found, breaking")
            break
        }
        time.Sleep(time.Second)
    }
    stream_file_bytes, err := ioutil.ReadFile("./files/" + file_name)
    if err != nil {
        fmt.Println(err)
        return
    }

    b := bytes.NewBuffer(stream_file_bytes)
    b.Next(1)
    b.WriteTo(w)
    fmt.Println("end")
}

func main() {
    http.HandleFunc("/files/", Zip)
    if err := http.ListenAndServe(":8090", nil); err != nil {
        panic(err)
    }
}