[英]How can you upload files as a stream in go?
There are a number of tutorials about posting files using http.Request<\/code> in go, but almost invariably they start like this:
有许多关于在 go 中使用
http.Request<\/code>发布文件的教程,但几乎总是这样开始:
file, err := os.Open(path)
if err != nil {
return nil, err
}
fileContents, err := ioutil.ReadAll(file)
If you need to set Content-Length
, it can be done manually.如果您需要设置
Content-Length
,可以手动完成。 The following snippet is an example of uploading file and extra parameters as a stream (the code based on Buffer-less Multipart POST in Golang )以下代码片段是将文件和额外参数作为流上传的示例(基于Golang 中的 Buffer-less Multipart POST 的代码)
//NOTE: for simplicity, error check is omitted
func uploadLargeFile(uri, filePath string, chunkSize int, params map[string]string) {
//open file and retrieve info
file, _ := os.Open(filePath)
fi, _ := file.Stat()
defer file.Close()
//buffer for storing multipart data
byteBuf := &bytes.Buffer{}
//part: parameters
mpWriter := multipart.NewWriter(byteBuf)
for key, value := range params {
_ = mpWriter.WriteField(key, value)
}
//part: file
mpWriter.CreateFormFile("file", fi.Name())
contentType := mpWriter.FormDataContentType()
nmulti := byteBuf.Len()
multi := make([]byte, nmulti)
_, _ = byteBuf.Read(multi)
//part: latest boundary
//when multipart closed, latest boundary is added
mpWriter.Close()
nboundary := byteBuf.Len()
lastBoundary := make([]byte, nboundary)
_, _ = byteBuf.Read(lastBoundary)
//calculate content length
totalSize := int64(nmulti) + fi.Size() + int64(nboundary)
log.Printf("Content length = %v byte(s)\n", totalSize)
//use pipe to pass request
rd, wr := io.Pipe()
defer rd.Close()
go func() {
defer wr.Close()
//write multipart
_, _ = wr.Write(multi)
//write file
buf := make([]byte, chunkSize)
for {
n, err := file.Read(buf)
if err != nil {
break
}
_, _ = wr.Write(buf[:n])
}
//write boundary
_, _ = wr.Write(lastBoundary)
}()
//construct request with rd
req, _ := http.NewRequest("POST", uri, rd)
req.Header.Set("Content-Type", contentType)
req.ContentLength = totalSize
//process request
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
} else {
log.Println(resp.StatusCode)
log.Println(resp.Header)
body := &bytes.Buffer{}
_, _ = body.ReadFrom(resp.Body)
resp.Body.Close()
log.Println(body)
}
}
Turns out you can actually pass the *File
(or any stream-like) object straight into NewRequest
.事实证明,您实际上可以将
*File
(或任何类似流的)对象直接NewRequest
给NewRequest
。
Notice the caveat however, that NewRequest (as shown here: https://golang.org/src/net/http/request.go?s=21674:21746#L695 ) won't actually set the ContentLength
unless the stream is explicitly one of:但是请注意,除非流明确,否则 NewRequest(如下所示: https ://golang.org/src/net/http/request.go?s=21674:21746#L695)实际上不会设置
ContentLength
其中之一:
Since *File
isn't one of these, the request will be sent without a content length unless you manually set it, which may cause some servers to discard the body of the incoming request, resulting in a body of ''
on the server when it appears to have been correctly sent from the go side.由于
*File
不是其中之一,除非您手动设置它,否则请求将在没有内容长度的情况下发送,这可能会导致某些服务器丢弃传入请求的正文,从而导致服务器上的正文为''
时它似乎是从 go 端正确发送的。
If the request must have a Content-Length
header (most file hosts reject upload requests without it), and you want to upload the file as a stream (without loading all to memory), standard library won't help you, and you have to calculate it yourself.如果请求必须具有
Content-Length
标头(大多数文件主机拒绝没有它的上传请求),并且您想将文件作为流上传(不将所有内容加载到内存中),标准库将无济于事,而您有自己计算。
Here's a minimal working example (without error checks) that uses io.MultiReader
to connect os.File
with other fields while keeping a tab on the request size.这是一个最小的工作示例(没有错误检查),它使用
io.MultiReader
将os.File
与其他字段连接,同时保留请求大小的选项卡。
It supports regular fields (with string content) and file fields, and calculates the total request body size.它支持常规字段(带有字符串内容)和文件字段,并计算总请求正文大小。 It's easy to extend it with other value types by simply adding a new
case
branch.只需添加一个新的
case
分支,就可以很容易地用其他值类型对其进行扩展。
import (
"crypto/rand"
"fmt"
"io"
"io/fs"
"mime"
"path/filepath"
"strings"
)
type multipartPayload struct {
headers map[string]string
body io.Reader
size int64
}
func randomBoundary() string {
var buf [8]byte
_, err := io.ReadFull(rand.Reader, buf[:])
if err != nil {
panic(err)
}
return fmt.Sprintf("%x", buf[:])
}
// Multipart request has the following structure:
// POST /upload HTTP/1.1
// Other-Headers: ...
// Content-Type: multipart/form-data; boundary=$boundary
// \r\n
// --$boundary\r\n 👈 request body starts here
// Content-Disposition: form-data; name="field1"\r\n
// Content-Type: text/plain; charset=utf-8\r\n
// Content-Length: 4\r\n
// \r\n
// $content\r\n
// --$boundary\r\n
// Content-Disposition: form-data; name="field2"\r\n
// ...
// --$boundary--\r\n
func prepareMultipartPayload(fields map[string]interface{}) (*multipartPayload, error) {
boundary := randomBoundary()
headers := make(map[string]string)
totalSize := 0
headers["Content-Type"] = fmt.Sprintf("multipart/form-data; boundary=%s", boundary)
parts := make([]io.Reader, 0)
CRLF := "\r\n"
fieldBoundary := "--" + boundary + CRLF
for k, v := range fields {
parts = append(parts, strings.NewReader(fieldBoundary))
totalSize += len(fieldBoundary)
if v == nil {
continue
}
switch v.(type) {
case string:
header := fmt.Sprintf(`Content-Disposition: form-data; name="%s"`, k)
parts = append(
parts,
strings.NewReader(header+CRLF+CRLF),
strings.NewReader(v.(string)),
strings.NewReader(CRLF),
)
totalSize += len(header) + 2*len(CRLF) + len(v.(string)) + len(CRLF)
continue
case fs.File:
stat, _ := v.(fs.File).Stat()
contentType := mime.TypeByExtension(filepath.Ext(stat.Name()))
header := strings.Join([]string{
fmt.Sprintf(`Content-Disposition: form-data; name="%s"; filename="%s"`, k, stat.Name()),
fmt.Sprintf(`Content-Type: %s`, contentType),
fmt.Sprintf(`Content-Length: %d`, stat.Size()),
}, CRLF)
parts = append(
parts,
strings.NewReader(header+CRLF+CRLF),
v.(fs.File),
strings.NewReader(CRLF),
)
totalSize += len(header) + 2*len(CRLF) + int(stat.Size()) + len(CRLF)
continue
}
}
finishBoundary := "--" + boundary + "--" + CRLF
parts = append(parts, strings.NewReader(finishBoundary))
totalSize += len(finishBoundary)
headers["Content-Length"] = fmt.Sprintf("%d", totalSize)
return &multipartPayload{headers, io.MultiReader(parts...), int64(totalSize)}, nil
}
then prepare the request, set the content length and send it:然后准备请求,设置内容长度并发送:
file, err := os.Open("/path/to/file.ext")
if err != nil {
return nil, err
}
defer file.Close()
up, err := prepareMultipartPayload(map[string]interface{}{
"a_string": "field",
"another_field": "yep",
"file": file, // you can have multiple file fields
})
r, _ := http.NewRequest("POST", "https://example.com/upload", up.body)
for k, v := range up.headers {
r.Header.Set(k, v)
}
r.ContentLength = up.size
c := http.Client{}
res, err := c.Do(r)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.