逐行读取 CSV 并将其解组为结构

Question

I have a 8 gigs CSV file that i need to unmarshal to a list of struct

package main

import (
    "encoding/csv"
    "fmt"
    "io"
    "os"

    gocsv "github.com/gocarina/gocsv"
    dto "github.com/toto/GeoTransport/import/dto"
)

// Put in parameter json the csv names
func importAdresse() {
    var adressesDB []dto.GeoAdresse
    clientsFile, err := os.OpenFile("../../../data/geo/public.geo_adresse.csv", os.O_RDWR|os.O_CREATE, os.ModePerm)
    if err != nil {
        panic(err)
    }
    gocsv.SetCSVReader(func(in io.Reader) gocsv.CSVReader {
        r := csv.NewReader(in)
        r.Comma = ';'
        return r // Allows use pipe as delimiter
    })
    if err = gocsv.UnmarshalFile(clientsFile, &adressesDB); err != nil { // Load clients from file
        panic(err)
    }
    var i int
    i = 0
    for _, adresse := range adressesDB {
        fmt.Println("adresse.Numero")
        fmt.Printf("%+v\n", adresse)
        fmt.Println(adresse.Numero)
        i++
        if i == 3 {
            break
        }
    }
}

func init() {
}

func main() {
    importAdresse()
}

Actually I am using go csv to unmarshall it but I have some memory error.

The program quit because it does not have enough ram.

I would like to know how to read the csv line by line and unmarshal it to a struct.

One of the solution will be to split the CSV file with some unix command.

But I would like to know how to do it with only Go.

Answer 1

It looks like the parsing method you're using attempts to read the entire CSV file into memory . You might try using the standard CSV reader package directly, or using another CSV-to-struct library that allows for line-by-line decoding like this one . Does the example code on those pages show what you're looking for?

Another thing to try would be running wc -l../../../data/geo/public.geo_adresse.csv to get the number of lines in your CSV file, then write this:

var adressesDB [<number of lines in your CSV>]dto.GeoAdresse

If the runtime raises the out of memory exception on that line, it means that the unmarshalled CSV data exceeds your RAM capacity and you'll have to read it in chunks.