Loop through huge CSV files

Question

I have to loop through a file with ~50.000 rows each day to generate reports and import those data records into our database

Since I have to execute some -replace -statements and stuff I currently loop through each row via foreach . This approach finishes after ~16 Minutes:

$csv_file = ".\testfile.csv"
$csv_import = Import-Csv $csv_file -Delimiter ";" -Encoding "default"

function Import-CsvVersion1 {
    $results = @()

    foreach ($data in $csv_import) {
        $properties = [ordered]@{
            id          = $data."id"
            name        = $data."name"
            description = $data."description"
            netcost     = $data."netcost"
            rrp         = $data."rrp"
        }
        $results += New-Object PSObject -Property $properties
    }

    # Export $results into final csv
}

I found another approach where the result of the foreach will be assigned directly to the $results variable. This approach finished after ~8 Minutes (so it needs only half of the time):

$csv_file = ".\testfile.csv"
$csv_import = Import-Csv $csv_file -Delimiter ";" -Encoding "default"

function Import-CsvVersion2 {
    $results = foreach ($data in $csv_import) {
        [PSCustomObject]@{
            id          = $data."id"
            name        = $data."name"
            description = $data."description"
            netcost     = $data."netcost"
            rrp         = $data."rrp"
        }
    }

    # Export $results into final csv
}

I've read somewhere that a loop via ForEach-Object may be even faster - unfortunately I don't know how to start with this.

Answer 1

Thanks to @GuentherSchmitz I was able to create the third test-function. In the next test I used a CSV-file with ~2.000 rows the results where:

• Import-CsvVersion1 -> 4 Minute(s) 24 Seconds
• Import-CsvVersion2 -> 0 Minute(s) 18 Seconds
• Import-CsvVersion3 -> 1 Minute(s) 20 Seconds

Thanks again for the help :-)

PS: I also got rid of a former Write-Progress which apparently slowed the script down by about 80%