Invoke-Webrequest 不同於瀏覽器
[英]Invoke-Webrequest different from browser
問題描述
我有一個腳本可以將 html 表解析為 powershell 表。 特定網站更新后腳本中斷，現在當我嘗試解析表格時，我得到錯誤信息（當我從 Internet Explorer 11 打開網站時也是如此）。 這是代碼：
獲取 HtmlTable
Function Get-HTMLTables{
<#
.SYNOPSIS Gets all the text in HTML tables on a webpage and returns it as an object of arrays
.DESCRIPTION
This function treats
1. table cells as properties of row objects
2. Rows as custom objects with cells as properties
3. Tables as arrays of rows
4. Pages as custom objects with numbered tables as properties.
Returns
-Custom object with numbered properties
-Each numbered property represents a table number from the top of the page down
-Each numbered property contains an array of row custom objects
-Each object in the array has properties representing each cell in the row.
.PARAMETER URL
The URL to look for HTML tables in
.PARAMETER FirstRowHeader
Whether or not to treat the first row of the table as a header
.EXAMPLE
$objPage = Get-HTMLTables -URL 'https://support.microsoft.com/en-us/lifecycle/search?sort=PN&alpha=windows%2010&Filter=FilterNO' -FirstRowHeader $true
.EXAMPLE
$objPage.21 | select -first 1
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Visual Studio Express 2015 for Windows 10
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes :
.EXAMPLE
$objPage.21 | Where{$_.('Products Released') -match 'Education'}
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Windows 10 Education, released in July 2015
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes : • Updates are cumulative, with each update built upon all of the updates that preceded it. A device needs to
install the latest update to remain supported.
• Updates may include new features, fixes (security and/or non-security), or a combination of both. Not all
features in an update will work on all devices.
• A device may not be able to receive updates if the device hardware is incompatible, lacking current
drivers, or otherwise outside of the Original Equipment Manufacturer’s (“OEM”) support period.
• Update availability may vary, for example by country, region, network connectivity, mobile operator (e.g.,
for cellular-capable devices), or hardware capabilities (including, e.g., free disk space).
#>
Param(
[uri]$URL = 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads',
[boolean]$firstRowHeader = $false
)
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
#Useragent
$userAgent = [Microsoft.PowerShell.Commands.PSUserAgent]::Safari
#Get the webpage
$page = Invoke-WebRequest $URL -UserAgent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
#Filter out only the tables
$tables = $page.ParsedHtml.body.getElementsByTagName('Table')
#Get only the tables that have cells
$tableswithcells = $tables | Where{$_.cells}
$hashPage = @{}
$tablecount = 0
#ForEach table
ForEach($table in $tableswithcells){
$arrTable = @()
$rownum = 0
$arrTableHeader = @()
#Get all the rows in the tables
ForEach($row in $table.rows){
#Treat the first row as a header
if($rownum -eq 0 -and $firstRowHeader){
ForEach($cell in $row.cells){
$arrTableHeader += $cell.InnerText.Trim()
}
#If not the first row, but using headers, store the value by header name
}elseIf($firstRowHeader){
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$strHeader = $arrTableHeader[$cellNum]
If($strHeader){
$hashRow.Add($strHeader,(($cell.innerhtml -replace "(?s)^.*$href=`"", '') -replace "`">Download</a>", ""))
#$hashRow.Add($strHeader,$cell.innertext)
}else{
#If the header is null store it by cell number instead
$hashRow.Add($cellnum,$cell.innertext)
}
$cellnum++
}
#Save the row as a custom ps object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
$arrTable += $objRow
#if not the first row and not using headers, store the value by cell index
}else{
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$hashRow.Add($cellnum,$cell.innertext)
$cellnum++
}
#Store the row as a custom object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
#Add the row to the array of rows
$arrTable += $objRow
}
$rownum++
}
#Add the tables to the hashtable of tables
$hashPage.Add($tablecount,$arrTable)
$tablecount++
}
$objPage = New-object -TypeName PSCustomObject -Property $hashPage
Return $objPage
}
當我執行以下幾行時：
$test = Get-HTMLTables -firstRowHeader $false -url 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$test.0
這些是結果：
5 : Windows x86-32
4 : Windows x86-64
3 : Mac OS X
2 : Linux x86-32
1 : Linux x86-64
0 : PostgreSQL Version
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 14.1
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 13.5
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 12.9
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 11.14
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 10.19
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.6.24*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.5.25*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.4.26*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.3.25*
我試圖抓取的網站： https://www.enterprisedb.com/downloads/postgres-postgresql-downloads ，您可以看到 Win64 的可用版本。
使用 Internet Explorer 11 打開此網站時也會發生同樣的情況，嘗試更改用戶代理但問題仍然存在。
1 個解決方案
解決方案1
0 2022-02-02 09:17:01
正如@Mathias評論的那樣：所需的（ Json ）數據在（ __NEXT_DATA__ ）腳本中。
要提取該信息，您需要執行以下操作：
$Page = Invoke-WebRequest 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$Unicode = [System.Text.Encoding]::Unicode.GetBytes($Page.Content)
$Document = New-Object -Com 'HTMLFile'
if ($Document.IHTMLDocument2_Write) { $Document.IHTMLDocument2_Write($Unicode) } else { $Document.write($Unicode) }
$Document.Close()
$Data = $Document.getElementById('__NEXT_DATA__').innerHTML |ConvertFrom-Json
$Data |ConvertTo-Json -Depth 10
產量：
{
"props": {
"pageProps": {
"postgreSQLDownloads": [
{
"version": "14.1",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "f027c016-7c5b-43fd-beb7-59ee43135607",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2016, Windows Server 2019",
"uuid": "db55e32d-e9f0-4d7c-9aef-b17d01210704",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
},
{
"version": "13.5",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "RPM",
"field_supported_os": "CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12",
"uuid": "514cbe6a-089c-4cb9-a6b1-ad9d6375842f",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "DEB",
"field_supported_os": "Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04",
"uuid": "e3c7350d-e18c-4eae-8817-c61cb83e09b9",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "76003af2-5f27-4b85-9804-c3b2237debf5",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2012 R2, Windows Server 2016, Windows Server 2019",
"uuid": "7c756686-90b4-4909-89ed-043e0705a76e",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
}
// ...
]
},
"__N_SSP": true
},
"page": "/downloads/postgres-postgresql-downloads",
"query": {},
"buildId": "u3IIRJVKDxUWWWroinlsJ",
"runtimeConfig": {},
"isFallback": false,
"gssp": true,
"scriptLoader": []
}
和
$Data.props.pageProps.postgreSQLDownloads.products |Format-Table
產量：
title field_installer_version field_sub_version field_os field_installation_method field_supported_os uuid field_show_this_version field_campaign_id field_product_category
----- ----------------------- ----------------- -------- ------------------------- ------------------ ---- ----------------------- ----------------- ----------------------
PostgreSQL Download 14 1 Mac OS X Interactive Installer MacOSX 10.12+ f027c016-7c5b-43fd-beb7-59ee43135607 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 14 1 Windows x86-64 Interactive Installer Windows Server 2016, Windows Server 2019 db55e32d-e9f0-4d7c-9aef-b17d01210704 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 514cbe6a-089c-4cb9-a6b1-ad9d6375842f 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 DEB Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04 e3c7350d-e18c-4eae-8817-c61cb83e09b9 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Mac OS X Interactive Installer MacOSX 10.12+ 76003af2-5f27-4b85-9804-c3b2237debf5 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 7c756686-90b4-4909-89ed-043e0705a76e 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 9a8d19d8-980c-4475-af6c-31aa964d2a30 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 DEB Debian 9.6.x, Ubuntu 18.04 0753b404-99a7-4c0d-867a-8e9534a566c7 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Mac OS X Interactive Installer MacOSX 10.12+ 942ce37d-8817-42be-8466-51f653728eda 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 03b13357-9281-4985-baea-17b72a0febc3 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 RPM ababa720-d9ac-46fa-88a0-161c27bdc4cf 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 & 32 DEB befdfd14-abe0-4614-b22b-0b4a577dfd7f 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Mac OS X Interactive Installer MacOSX 10.12+ dcf30630-6453-4c50-9de1-add7a8bf431c 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 995d0688-d8cf-48b8-8afe-13c777b0ef8e 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 10 19 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 f6607ede-6698-4116-bf56-bd25284203aa 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, Ubuntu 14.04 beb6b65f-6aeb-466f-8b9f-c27a7e327ccf 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 RPM 03f51b70-24ee-4c75-88a7-de2e488c59da 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 & 32 DEB a92ccaab-7c3f-4a5b-9a5e-e632b79b33d7 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Mac OS X Interactive Installer MacOSX 10.12+ c1b50eb7-9ad6-4e6c-a094-9a7f70ee86f9 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-32 Interactive Installer Windows 7 / 8 / 10 e9a60157-955a-4f24-a738-07cff51ccb43 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-64 Windows 2012 R2 / R1, Windows 7 / 8 / 10, Windows Server 2016 ea5c8104-3940-4ed1-b427-81cf19781581 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 a9b88ba3-839b-4883-b2c5-378d4a7ee3da 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, SLES 12, Ubuntu 14.04 c539f198-0912-46a9-b4e6-545166574893 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 RPM ceb7a789-4a0c-4c17-8d48-abb1cdfd9cff 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 & 32 DEB b976efbe-41d1-4d3f-9b14-95b7b03e651e 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Mac OS X Interactive Installer MacOSX 10.12+ b40e8d78-4a3b-458a-85f5-f0b7e0bbfa31 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 5011320c-34df-4c8f-9b6f-21e53c4a4930 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1, Windows 7 / 8 / 10 91f8264f-8a2d-434a-9559-4dfa9b3b80b1 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 25ec7179-be39-4e0f-a429-bc140ccfab00 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 Interactive Installer Debian 7, SLES 12, Ubuntu 14.04 147a6582-849c-44d1-8a51-e90fcece30a6 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 RPM 5310449a-13b3-4ca1-922c-f0baef05c3ab 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 & 32 DEB 654437d3-3c63-449c-bd22-1deda40dad16 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Mac OS X Interactive Installer MacOSX 10.10, MacOSX 10.8, MacOSX 10.9 3a9e2080-a11f-4230-8974-78c2ef7234d9 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 aa5aa60c-2030-4e06-a6ba-06ba0eb79383 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1 7869632c-134f-4284-8fba-feb67010b372 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-32 Interactive Installer ed2f791f-ca94-4872-ac52-d3221e1c1950 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-64 Interactive Installer 5273bfc5-9382-4a55-ab9a-d42c86296dac 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Mac OS X Interactive Installer 58b175c3-0c86-425a-a3fa-5bbdb10abf08 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-32 Interactive Installer 33e923ed-0d9e-40b1-8972-96c245d239e4 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-64 Interactive Installer cba59404-da4d-4009-8edb-6655b72464dd 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-32 Interactive Installer 0e139bf2-f0da-4ba9-bb40-9c0cadb4087e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-64 Interactive Installer 566673e2-7d5e-46aa-82dc-7400d296400e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Mac OS X Interactive Installer 72c6d5f4-68bb-4080-a8cf-769f5fc60f41 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-32 Interactive Installer 46c086f2-944c-49a2-bac3-204ccf14ac17 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-64 Interactive Installer dbb5a7ac-4ecb-4a11-be2b-d56e8908b2a4 0 70150000000rxXPAAY PostgreSQL
Invoke-WebRequest 和 Invoke-RestMethod 的不同結果
[英]Different results from Invoke-WebRequest and Invoke-RestMethod
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.
相關問題
Invoke-WebRequest從瀏覽器產生不同的結果
Invoke-WebRequest 和 Invoke-RestMethod 的不同結果
在Powershell中格式化Invoke-WebRequest的輸出
如何解析來自 Invoke-WebRequest 的響應
Invoke-WebRequest接受來自stdin的正文
將 Invoke-WebRequest 的輸出流式傳輸到屏幕
來自Invoke-WebRequest響應的XML解析
檢查來自invoke-webrequest的內容以查找字符串
調用WebRequest超時
使用PScredentials調用WebRequest
相關標簽