Download file from URL if not modified recently using Powershell

Question

I have a Powershell script that is downloading files continuously from a URL public directory. The script is supposed to download files from the directory that does not currently exist in our local directory. The script looks like this:

$outputdir = 'C:\mydir\public\demos'
$url       = 'https://xxx.xxx.net/fastdl/xxx/xxx/public/'

# enable TLS 1.2 and TLS 1.1 protocols
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12, [Net.SecurityProtocolType]::Tls11

$WebResponse = Invoke-WebRequest -Uri $url
# get the list of links, skip the first one ("../") and download the files
$WebResponse.Links | Select-Object -ExpandProperty href -Skip 0 | ForEach-Object {
    Write-Host "Downloading file '$_'"
    $filePath = Join-Path -Path $outputdir -ChildPath $_
    $fileUrl  = '{0}/{1}' -f $url.TrimEnd('/'), $_
    
    if (Test-Path($filePath)) 
    {
        Write-Host 'Skipping file, already downloaded' -ForegroundColor Yellow
        return
    }
    Invoke-WebRequest -Uri $fileUrl -OutFile $filePath
}

Now, since I want to download all files from the $url directory I need to check one more condition. The file in the $url can't be modified in the last 10 minutes, if it has then the script should skip the file and download it once the last modification time is greater than 10 minutes.

I have not found any syntax that seem to be able to fit an if statement like that. Any ideas?

Answer 1

I would change Test-Path to Get-Item so if that returns something, the file exists and you can examine its LastWriteTime property:

$WebResponse.Links | Select-Object -ExpandProperty href -Skip 0 | ForEach-Object {
    $filePath = Join-Path -Path $outputdir -ChildPath $_
    $fileUrl  = '{0}/{1}' -f $url.TrimEnd('/'), $_

    # set up a boolean fag to download or not
    $downloadThis = $true  
    if ($file = Get-Item -Path $filePath -ErrorAction SilentlyContinue) {
        # $file exists, check the LastWriteTime property
        if ($file.LastWriteTime -ge (Get-Date).AddMinutes(-10)) {
            Write-Host 'Skipping file, already downloaded' -ForegroundColor Yellow
            $downloadThis = $false
        }
    }
    if ($downloadThis) {
        Write-Host "Downloading file '$($file.Name)'"
        Invoke-WebRequest -Uri $fileUrl -OutFile $filePath
    }
}

Answer 2

I think all you need is to calculate time from now to the last modified time of the file. I once write a function to check a file created less than 10 mins or more than 10 mins and action accordingly, Hope fully it will help

#function to check whether a file is more than 10 mins age, if yes, return TRUE. If not, return FALSE
function file_age ($file){
#$file_info = Get-Item $file
$createtime = $file.CreationTime
$now = get-date

if (($now - $createtime).totalminutes -ge 10) {  
    Write-host "file [$file] created equal or more than 10 mins ago"
    return $true
    }
else
    {#Write-host  "file [$file] created within the 10 mins"
    return $false
    }
}