Powershell filedownload from https

Question

I am trying to download several files using Powershell and its Invoke-WebRequest method.

I'm basically looping through several filenames (I know that they are available on the server) and download them.

My problem is that my script works for the first file and fails for every file that follows. When I open one of the later files (.csv`s) there is just some html code in it).

I already read a lot about passing session cookings but I am not sure if this is my problem or how I can do that.

My script so far looks like this:

$httpsUser = 'XXX'
$httpsPass = 'YYY'

foreach ($instrument in 'ivv','ijh','ijr','iwm') {

$Source = 'https://***', `
$instrument, '-en_us.csv' -join ""

$Target = 'C:\User\', `
$instrument, '-en_us.csv' -join ""

$uri = New-Object “System.Uri” “$Source”
$WebClient = [System.Net.HttpWebRequest]::Create($uri) 
$webclient.Proxy.Credentials =
[System.Net.CredentialCache]::DefaultNetworkCredentials
$webclient.Credentials =
New-Object System.Net.NetworkCredential($httpsUser,$httpsPass)

Invoke-WebRequest -Uri $Source -OutFile $Target
}

Thank you all and let me know what you think :)

Answer 1

It seems like you don't using the HttpWebRequest you created to download the file. Anyway, I would recommend using System.Net.WebClient :

$wc = New-Object System.Net.WebClient
$wc.Credentials =  New-Object System.Net.NetworkCredential($httpsUser,$httpsPass)
$wc.DownloadFile($Source, $target)

Answer 2

Try using webclient methods DownloadFile(src,dst). Should be something like this:

$httpsUser = 'XXX'
$httpsPass = 'YYY'

foreach ($instrument in 'ivv','ijh','ijr','iwm') {

$Source = 'https://***', `
$instrument, '-en_us.csv' -join ""

$Target = 'C:\User\', `
$instrument, '-en_us.csv' -join ""

$webclient = New-Object -TypeName Net.WebClient
$webclient.Encoding = [System.Text.Encoding]::UTF8
$webclient.UseDefaultCredentials = $true
$webclient.Proxy.Credentials = New-Object System.Net.NetworkCredential($httpsUser,$httpsPass)
$webclient.DownloadFile($Source,$Target)
}

Answer 3

Thank you all for your answers. I figured out now that the site generates a security token while logging in. This token needs to be passed in every webrequest. I was not yet able to figure out how to do that with powershell but know that perl has a build in function (called $merch) for exactly this problem.

To solve my problem I had to automatize the IE :( (I know this is not the most sophisticated way but right now it is the quickest solution. If anyone is interested here is the code for that:

$ie = new-object -ComObject 'InternetExplorer.Application'
$requestUri = 'https://www.trololo.com'
$userIdFragment = "userName";
$passwordIdFragment = "password";
$buttonIdFragment = "submitLogin";
$ie.visible = 'false'


$ie.navigate($requestUri)
while($ie.Busy) { Start-Sleep -Milliseconds 100 }


$doc1 = $ie.Document
$doc1.getElementsByTagName("input") | % {
    if ($_.id -ne $null){
        if ($_.id.Contains($buttonIdFragment)) { $btn = $_ }
        if ($_.id.Contains($passwordIdFragment)) { $pwd = $_ }
        if ($_.id.Contains($userIdFragment)) { $user = $_ }
    }
}

$user.value = "XXXX"
$pwd.value = "YYYY
$btn.disabled = $false
$btn.click()
while($ie.Busy) { Start-Sleep -Milliseconds 5000 }
$ie.navigate($requestUri)
while($ie.Busy) { Start-Sleep -Milliseconds 200 }
$doc1 = $ie.Document


$link = $doc1.getElementsByTagName("a") | where-object {$_.href -match "Your String"}
$link.click()
Start-Sleep -Milliseconds 1000
$wshell = new-object -com wscript.shell
$wshell.appactivate("Internet Explorer")
$wshell.sendkeys("%s")