How to pass io.Reader instance to a function in Golang?

Question

I've been dealing with an issue where I have to put an io.Reader instance as a parameter to a function provided as an end-point by an api. The task I need to do is to upload local folder to company's cloud storage.

func (s *server) uploadFileToPutIo(w http.ResponseWriter, r *http.Request) {
    tokenSource := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token})
    oauthClient := oauth2.NewClient(context.TODO(), tokenSource)
    client := putio.NewClient(oauthClient)

    var testIO io.Reader // ? 

    upload, err := client.Files.Upload(context.TODO(), testIO, "test", 0)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(upload.File)
    sendResponse(w, []byte("successful"), http.StatusOK)
}

When I make a request to this end-point /upload under POST method. I get the following error.

2021/12/01 18:28:47 http: panic serving 127.0.0.1:61057: runtime error: invalid memory address or nil pointer dereference
goroutine 8 [running]:
net/http.(*conn).serve.func1(0xc000108d20)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
        /usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026360, 0x0, 0x0, 0xc000170000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
        /usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
        /usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc000010088, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
        /Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio@v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a2a0, 0xc000154500)
        /Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a2a0, 0xc000154500)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
        /Users/barisertas/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108d20, 0x1468400, 0xc00005e300)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
net/http.(*conn).serve.func1(0xc000108dc0)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
        /usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026400, 0x0, 0x0, 0xc000198000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
        /usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
        /usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc0000100c8, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
        /Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio@v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a380, 0xc000154800)
        /Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a380, 0xc000154800)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a380, 0xc000154600)
        /Users/barisertas/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a380, 0xc000154600)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108dc0, 0x1468400, 0xc00005e580)
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
        /usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b

And this is the documentation of the function I am trying to use:

// Upload reads from given io.Reader and uploads the file contents to Put.io
// servers under directory given by parent. If parent is negative, user's
// preferred folder is used.
//
// If the uploaded file is a torrent file, Put.io will interpret it as a
// transfer and Transfer field will be present to represent the status of the
// tranfer. Likewise, if the uploaded file is a regular file, Transfer field
// would be nil and the uploaded file will be represented by the File field.
//
// This method reads the file contents into the memory, so it should be used for
// <150MB files.
func (f *FilesService) Upload(ctx context.Context, r io.Reader, filename string, parent int64) (Upload, error) {
    if filename == "" {
        return Upload{}, fmt.Errorf("filename cannot be empty")
    }

    var buf bytes.Buffer
    mw := multipart.NewWriter(&buf)

    // negative parent means use user's preferred download folder.
    if parent >= 0 {
        err := mw.WriteField("parent_id", itoa(parent))
        if err != nil {
            return Upload{}, err
        }
    }

    formfile, err := mw.CreateFormFile("file", filename)
    if err != nil {
        return Upload{}, err
    }

    _, err = io.Copy(formfile, r)
    if err != nil {
        return Upload{}, err
    }

    err = mw.Close()
    if err != nil {
        return Upload{}, err
    }

    req, err := f.client.NewRequest(ctx, "POST", "/v2/files/upload", &buf)
    if err != nil {
        return Upload{}, err
    }
    req.Header.Set("Content-Type", mw.FormDataContentType())

    var response struct {
        Upload
    }
    _, err = f.client.Do(req, &response)
    if err != nil {
        return Upload{}, err
    }
    return response.Upload, nil
}

I am confused about putting the io.Reader instance to a function. How can I put the io.Reader function properly under this case. Just an instance after creating as follows: var testIO io.Reader or should I do some extra operations?

Answer 1

runtime error: invalid memory address or nil pointer dereference

As you surely know, this is because you've declared an io.Reader but you haven't set its value, so it is still equal to the default value of an interface, which is nil .

    var testIO io.Reader // ? 

The point of passing the io.Reader to Upload is to provide the data to be uploaded. By passing an io.Reader , an arbitrary source of data can provide an abitrary quantity of bytes, unrestricted by memory availability (unlike []byte , which would require holding all data in memory before uploading). io.Reader is commonly used for providing data for this kind of "streaming" operation.

Upload reads from given io.Reader and uploads the file contents

That io.Reader should be the source of the data to upload.

io.Reader could be a file from os.Open() .

But it could be anything that satisfies io.Reader - for example, it could also be a bytes.Buffer .

It could even be something more esoteric, like the result of an GetObject API call against the popular S3 service from AWS , which also returns an io.ReadCloser which satisfies io.Reader .

io.Reader is a great example of how Go interfaces allow independent libraries to be connected to each other. the SDK you're using doesn't care what io.Reader it's passed; it is enough that the value satisfies io.Reader , a requirement enforced at compile time. You can pass it anything that satisfies io.Reader , and the interface type guarantees that Upload() will be able to handle it properly.

Upload takes an io.Reader . If you want to pass it something like an *os.File from os.Open or an io.ReadCloser from say S3 GetObject, that works because io.ReadCloser satisfies io.Reader . But since Upload takes io.Reader not io.ReadCloser you can be confident that it will call nothing but the functions defined in that interface. You'll have to do any closing yourself, after Upload is called.

Make sure to take the time to understand how io.Reader leaves the input to this function open-ended while also being specific about the interface it expects. This is one of the most important concepts in Go.

Answer 2

This:

var testIO io.Reader

is equivalent to this:

testIO := io.Reader(nil)

so this is why you are getting a panic of a nil-pointer reference:

2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:

An io.Reader is an interface which allows one to pass generic values, provided they implement the interface (ie implement the Read method).

Since you are uploading a file, your byte stream should come from an OS file. os.File implements the correct Read method - so is compatible io.Reader .

So try:

f, err := os.Open(uploadFilePath)
if err != nil { /* ... */ }

upload, err := client.Files.Upload(context.TODO(), f, "test", 0)

Answer 3

When you define a variable, it is initial value is the zero value for that type. io.Reader is an interface and the zero value for it is nil . Thus the nil pointer dereference error . Simply initialize the io.Reader before passing it to Upload:

file, err := os.Open("path/to/file")
// if err != nil { ... }

upload, err := client.Files.Upload(context.TODO(), file, "test", 0)