Optimizing PSObject creation

Question

As with the code below, it would take about a hour to process the script. I am trying to find ways to lessen the time as this script will be scheduled to run every day.

I noticed that when creating new object, it would use same $_.Tag.Substring 1440 times for each tag, and there are over 200 tags.

Also, Measure-Object is used 3 times to process through the same $_.Group to get min, average, and max value.

How do I set them up in a way it doesn't do those same processes over again?

$data = Get-ChildItem $destinationPath\PPP_*.csv | 
    ForEach-Object { $_.FullName } | 
    Import-Csv -Header Tag, Value, Date, Time, Mode, Alarm

$data | ForEach-Object {
    [pscustomobject]@{
        "Column 1"  = $_.Tag.Substring(8,2)
        "Pump Unit" = $_.Tag.Substring(10,2)
        "Column 3"  = $_.Tag.Substring(12,2)
        "Column 4"  = $_.Tag.Substring(14,1)
        "Value"     = $_.Value
        "Time"      = $_.Time
    }
} | Export-Csv -Path "$destinationPath\Summary.csv" -NoTypeInformation

$data | Group-Object Tag | ForEach-Object {
    [pscustomobject]@{
        "Column 1"  = $_.Name.Substring(8,2)
        "Pump Unit" = $_.Name.Substring(10,2)
        "Column 3"  = $_.Name.Substring(12,2)
        "Column 4"  = $_.Name.Substring(14,1)
        "Minimum"   = ($_.Group | Measure-Object -Property Value -Minimum).Minimum
        "Average"   = ($_.Group | Measure-Object -Property Value -Average).Average
        "Maximum"   = ($_.Group | Measure-Object -Property Value -Maximum).Maximum
    }
} | Export-Csv -Path "$destinationPath\Statistics.csv" -NoTypeInformation

Answer 1

Try this. I believe this requires PowerShell v3 or higher for the Add-Member generated properties to "stick" to the $data elements.

$data = $data | ForEach-Object {
  Add-Member -InputObject $_ NoteProperty -Name "Column 1"  -Value ($_.Tag.Substring(8,2))
  Add-Member -InputObject $_ NoteProperty -Name "Pump Unit" -Value ($_.Tag.Substring(10,2))      
  Add-Member -InputObject $_ NoteProperty -Name "Column 3"  -Value ($_.Tag.Substring(12,2))
  Add-Member -InputObject $_ NoteProperty -Name "Column 4"  -Value ($_.Tag.Substring(14,1))
  $_
}

$data | ForEach-Object {
  [pscustomobject]@{
    "Column 1"  = $_."Column 1"
    "Pump Unit" = $_."Pump Unit"
    "Column 3"  = $_."Column 3"
    "Column 4"  = $_."Column 4"
    "Value"     = $_.Value
    "Time"      = $_.Time
  }
} | Export-Csv -Path "$destinationPath\Summary.csv" -NoTypeInformation

To avoid duplication of Measure-Object change to this:

$data | Group-Object Tag | ForEach-Object {
  $measInfo = $_.Group | Measure-Object -Property Value -Minimum -Maximum -Average
  [pscustomobject]@{
    "Column 1"  = $_.Group[0]."Column 1"
    "Pump Unit" = $_.Group[0]."Pump Unit"
    "Column 3"  = $_.Group[0]."Column 3"
    "Column 4"  = $_.Group[0]."Column 4"
    "Minimum"   = $measInfo.Minimum
    "Average"   = $measInfo.Average
    "Maximum"   = $measInfo.Maximum
  }
} | Export-Csv -Path "$destinationPath\Statistics.csv" -NoTypeInformation

Answer 2

I modified it a bit from Keith's code so that everything processes from one ForEach-Object :

 Get-ChildItem $destinationPath\PPP_*.csv | 
    ForEach-Object { $_.FullName } | 
    Import-Csv -Header Tag, Value, Date, Time, Mode, Alarm | 
    Group-Object -Property Tag | 
    ForEach-Object {

        $tag        = $_.Name
        $column1    = $tag.Substring(8,2)
        $pumpUnit   = $tag.Substring(10,2)
        $column3    = $tag.Substring(12,2)
        $column4    = $tag.Substring(14,1)

        $_.Group | ForEach-Object {
            [pscustomobject]@{
                "Column 1"  = $column1
                "Pump Unit" = $pumpUnit
                "Column 3"  = $column3
                "Column 4"  = $column4
                "Value"     = $_.Value
                "Time"      = $_.Time
            }
        } | Export-Csv -Path "$destinationPath\Summary.csv" -Append -NoTypeInformation


        $measInfo = $_.Group | Measure-Object -Property Value -Minimum -Average -Maximum

        [pscustomobject]@{
            "Column 1"  = $column1
            "Pump Unit" = $pumpUnit
            "Column 3"  = $column3
            "Column 4"  = $column4
            "Minimum"   = $measInfo.Minimum
            "Average"   = $measInfo.Average
            "Maximum"   = $measInfo.Maximum
        } | Export-Csv -Path "$destinationPath\Statistics.csv" -Append -NoTypeInformation

    }