Copying new files from newly created folders in my S3 bucket to my local machine using PowerShell

Question

We have a program running on a server that copies .xml files to our S3 bucket. But it creates folders according to the date. So a folder for the year is created and then a sub-folder for the month is created, and then yet another sub-folder for the day. For example:

S3://AppMyBucket/Documents/2018/Oct/05

So every day it's creating a folder in accordance with the day, 01 , 02 , 03 , etc., and putting .xml files in those folders. I have to create a PowerShell script to run every 15 minutes to copy new files that go into the latest folder created from our S3 bucket to a folder on my local machine and then call another program to process those files.

My question is how can I copy new files created from new folders created in our S3 bucket ( S3://AppBucket/Documents/2018/Oct/05... ) to my local Windows machine ( D:\\DocFolder\\ ) and then call program (AIMParser.exe) to process those .xml files?

Please forgive me as I am a beginner at PowerShell.

Answer 1

The following example:

• Uses WinSCP .NET assembly to access the S3 bucket
• Finds the latest year (lexicographically)
• Finds the latest month in the year
• Finds the latest day in the month (lexicographically)
• Finds the latest file in the day (by file timestamp)

Add-Type -Path "WinSCPnet.dll"

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::S3
    HostName = "s3.amazonaws.com"
    PortNumber = 443
    UserName = "access key id"
    Password = "secret access key"
}

Write-Host "Connecting..."
$session = New-Object WinSCP.Session
$session.SessionLogPath = "s3.log"
$session.Open($sessionOptions)

$remotePath = "/AppMyBucket/Documents";
$localPath = "D:\DocFolder"

$yearFolder =
    $session.ListDirectory($remotePath).Files |
        Where-Object { $_.IsDirectory } |
        Sort-Object -Descending |
        Select-Object -First 1
Write-Host "Latest year is $($yearFolder.Name)"

$yearPath = [WinSCP.RemotePath]::Combine($remotePath, $yearFolder.Name)

$monthFolder =
    $session.ListDirectory($yearPath).Files |
        Where-Object { $_.IsDirectory } |
        Sort-Object -Descending -Property @{ Expression = {
            Get-Date -Date "$($_.Name) 1 $($yearFolder.Name)" -Format "MM" } } |
        Select-Object -First 1

Write-Host "Latest month is $($monthFolder.Name)"

$monthPath = [WinSCP.RemotePath]::Combine($yearPath, $monthFolder.Name)

$dayFolder =
    $session.ListDirectory($monthPath).Files |
        Where-Object { $_.IsDirectory } |
        Sort-Object -Descending |
        Select-Object -First 1

Write-Host "Latest day is $($dayFolder.Name)"

$dayPath = [WinSCP.RemotePath]::Combine($monthPath, $dayFolder.Name)

$latest = 
    $session.ListDirectory($dayPath).Files |
        Where-Object { -Not $_.IsDirectory } |
        Sort-Object LastWriteTime -Descending |
        Select-Object -First 1

Write-Host "Latest file is $($latest.Name), downloading..."

$session.GetFiles($latest.FullName, (Join-Path $localPath "*")).Check()

Write-Host "Done"
$session.Dispose()

(I'm the author of WinSCP)