Powershell Export csv from for loop

Question

The following code gets a file from FTP server, stores it in a FTP location, then loops through each record and removes duplicates. Finally, exports csv. All this is working fine except it is taking long for the final csv to get exported. Is there a better way to code in order to make the process faster?

 
Get-SFTPFile -SFTPSession $Session -RemoteFile "/stms/inbound/Working_Titles_Primary.csv" -LocalPath "$env:TEMP"  

if ($? -eq $false) {return}


 $WT = import-csv "$env:TEMP\Working_Titles_Primary.csv"  
 
 if ((Test-Path -Path "C:\Scripts_Dev\MasterScript\WorkingTitles.csv") -eq $True) {Remove-Item "C:\Scripts_Dev\MasterScript\WorkingTitles.csv" -Force}
 remove-Item "$env:TEMP\Working_Titles_Primary.csv" -Force #Removes file from Temp folder
 $noDupes = @()
 

 $Output =   foreach ($aa in $WT)
   {

        if($aa.PF_USER_ID -in $noDupes)
        {
         
        }

        else{
        
          New-Object -TypeName PSObject -Property @{

                OLO_CODE  = $aa.OLO_CODE 
                PF_USER_ID = $aa.PF_USER_ID
                ACTIVE_INDICATOR = $aa.ACTIVE_INDICATOR
                LAST_DAY_EMPLOYED = $aa.LAST_DAY_EMPLOYED
                APPT_DATE = $aa.APPT_DATE
                EMPLOYEE_TYPE = $aa.EMPLOYEE_TYPE
                FIRST_NAME = $aa.FIRST_NAME
                LAST_NAME = $aa.LAST_NAME
  } | Select-Object OLO_CODE,PF_USER_ID,ACTIVE_INDICATOR,EMPLOYEE_TYPE,FIRST_NAME,LAST_NAME | Export-csv -Path "C:\Scripts_Dev\MasterScript\WorkingTitles.csv" -Append 
        }
      $noDupes += ($aa.PF_USER_ID)
       
    }

Answer 1

Following Lee_Dailey's comment, here is how you can make your code run faster.

Get-SFTPFile -SFTPSession $Session -RemoteFile "/stms/inbound/Working_Titles_Primary.csv" -LocalPath "$env:TEMP"  

if (-not $?) {break}

$WT = import-csv "$env:TEMP\Working_Titles_Primary.csv"  
 
if (Test-Path -Path "C:\Scripts_Dev\MasterScript\WorkingTitles.csv")
{
    Remove-Item "C:\Scripts_Dev\MasterScript\WorkingTitles.csv" -Force
}

Remove-Item "$env:TEMP\Working_Titles_Primary.csv" -Force #Removes file from Temp folder

$index = [system.collections.generic.list[string]]::new()
$output = [system.collections.generic.list[pscustomobject]]::new()

foreach ($i in $WT)
{
    $z = $i.PF_USER_ID.ToLower()

    if($index.contains($z))
    {
        continue
    }

    $index.Add($z)

    $output.Add(
        [pscustomobject]@{
            OLO_CODE  = $i.OLO_CODE 
            PF_USER_ID = $i.PF_USER_ID
            ACTIVE_INDICATOR = $i.ACTIVE_INDICATOR
            LAST_DAY_EMPLOYED = $i.LAST_DAY_EMPLOYED
            APPT_DATE = $i.APPT_DATE
            EMPLOYEE_TYPE = $i.EMPLOYEE_TYPE
            FIRST_NAME = $i.FIRST_NAME
            LAST_NAME = $i.LAST_NAME
    }) 
}

if($output)
{
    $output|Export-csv -Path "C:\Scripts_Dev\MasterScript\WorkingTitles.csv"
}