如何动态获取 Go 中的文件长度？

Question

I have the following code snippet:

func main() {
    // Some text we want to compress.
    original := "bird and frog"
    
    // Open a file for writing.
    f, _ := os.Create("C:\\programs\\file.gz")
    
    // Create gzip writer.
    w := gzip.NewWriter(f)
    
    // Write bytes in compressed form to the file.
    while ( looping over database cursor) {
       w.Write([]byte(/* the row from the database as obtained from cursor */))
    }
    
    // Close the file.
    w.Close()
    
    fmt.Println("DONE")
}

However, I wish to know a small modification. When the size of file reaches a certain threshold I want to close it and open a new file. And that too in compressed format.

For example:

Assume a database has 10 rows each row is 50 bytes.

Assume compression factor is 2, ie 1 row of 50 bytes is compressed to 25 bytes.

Assume a file size limit is 50 bytes.

Which means after every 2 records I should close the file and open a new file.

How to keep track of the file size while its still open and still writing compressed documents to it?

Answer 1

gzip.NewWriter takes a io.Writer . It is easy to implement custom io.Writer that does what you want.

Eg Playground

type MultiFileWriter struct {
    maxLimit      int
    currentSize   int
    currentWriter io.Writer
}

func (m *MultiFileWriter) Write(data []byte) (n int, err error) {
    if len(data)+m.currentSize > m.maxLimit {
        m.currentWriter = createNextFile()
    }
    m.currentSize += len(data)
    return m.currentWriter.Write(data)
}

Note: You will need to handle few edge cases like what if len(data) is greater than the maxLimit . And may be you don't want to split a record across files.

Answer 2

You can use the os.File.Seek method to get your current position in the file, which as you're writing the file will be the current file size in bytes.

For example:

package main

import (
    "fmt"
    "os"
)

func main() {
    words := []string{"this", "is", "a", "test"}

    fd, err := os.Create("testfile.txt")
    if err != nil {
        panic(err)
    }
    defer fd.Close()

    for _, word := range words {
        fd.Write([]byte(word))
        fd.Write([]byte(" "))
        pos, err := fd.Seek(0, os.SEEK_CUR)
        if err != nil {
            panic(err)
        }

        fmt.Printf("file position: %d\n", pos)
    }
}

Which outputs:

file position: 5
file position: 8
file position: 10
file position: 15

And we can confirm with wc :

$ wc -c testfile.txt
15 testfile.txt