在 HDInsight 上下载 Hive 输出时未找到 WindowsAzure 程序集

Question

以下脚本从运行 Windows 8.1 的本地计算机运行配置单元作业，并将输出复制到本地文件。 到目前为止，它一直运行良好，但升级到 Windows 10 后，当我调用Get-AzureHDInsightJobOutput 。 我曾尝试卸载 Windows Azure Powershell 并重新安装，但这并没有帮助。 不知道现在如何进行，有人有任何想法吗？

# Retrieve a single tagged stream from project hive database into a single file
#
#
param(
    [string]$table = 'sensor_part',
    [string]$bld = 'B24',
    [string]$tag = 'B24/AHU/LR/_1/OAT'
)

$clusterName = 'Crunch88'
$HiveQuery = "select * from ${table} where bld='${bld}' and tag='${tag}'"
$jobDefinition = New-AzureHDInsightHiveJobDefinition -Query $HiveQuery -StatusFolder '/tagtransfer'

$job = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $jobDefinition
Write-Host "Wait for the Hive job to complete ..." -ForegroundColor Green
Wait-AzureHDInsightJob -Job $job -WaitTimeoutInSeconds 72000

Get-AzureHDInsightJobOutput -StandardError -JobId $job.JobId -Cluster $clusterName
Write-Host "Copy the standard output to a local file" -ForegroundColor Green

Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $job.JobId -DownloadTaskLogs -TaskLogsDirectory "tasklogs"

# AzCopy /Y /Source:${blobcontainer} /Dest:${tsvpath} /SourceKey:${blobkey} /Pattern:${dirout}/stdout 

这是带有错误消息的运行结果：

PS C:\Windows\System32\WindowsPowerShell\v1.0> F:\88acres\3-hive\hiveretrieval\error.ps1
WARNING: When submitting a query use the -RunAsFile switch to prevent errors with query lengths or special characters
Wait for the Hive job to complete ...


Cluster         : Crunch88
ExitCode        : 0
Name            : Hive: select * from sensor
PercentComplete : map 100% reduce 0%
Query           : select * from sensor_part where bld='B24' and tag='B24/AHU/LR/_1/OAT'
State           : Completed
StatusDirectory : /tagtransfer
SubmissionTime  : 30-Aug-15 12:26:18
JobId           : job_1438448805059_0015

Get-AzureHDInsightJobOutput : Could not load file or assembly 'Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, 
PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At F:\88acres\3-hive\hiveretrieval\error.ps1:18 char:1
+ Get-AzureHDInsightJobOutput -StandardError -JobId $job.JobId -Cluster ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-AzureHDInsightJobOutput], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.GetAzu 
   reHDInsightJobOutputCmdlet

Copy the standard output to a local file
Get-AzureHDInsightJobOutput : Could not load file or assembly 'Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, 
PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At F:\88acres\3-hive\hiveretrieval\error.ps1:21 char:1
+ Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $job.JobId - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-AzureHDInsightJobOutput], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.GetAzu 
   reHDInsightJobOutputCmdlet

更新：我已启用融合日志记录。 结果表明它绑定到 5.0.0.0 而不是 3.0.3.0。 不知道如何解决这个问题。

3.0.3.0 的失败日志条目如下：

*** Assembly Binder Log Entry  (04-Sep-15 @ 08:54:11) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable  C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: DisplayName = Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///C:/Windows/System32/WindowsPowerShell/v1.0/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = powershell_ise.exe
Calling assembly : Microsoft.Hadoop.Client, Version=1.5.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage.DLL.
LOG: Assembly download was successful. Attempting setup of file: C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\HDInsight\Microsoft.WindowsAzure.Storage.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
WRN: Comparing the assembly name resulted in the mismatch: Major Version
ERR: The assembly reference did not match the assembly definition found.
ERR: Run-from-source setup phase failed with hr = 0x80131040.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.EXE.
LOG: All probing URLs attempted and failed.


*** 

Assembly Binder Log Entry  (04-Sep-15 @ 08:54:11) ***

这是 5.0.0.0 的成功版本：

*** Assembly Binder Log Entry  (04-Sep-15 @ 08:54:10) ***

The operation was successful.
Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable  C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: DisplayName = Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///C:/Windows/System32/WindowsPowerShell/v1.0/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = powershell_ise.exe
Calling assembly : Microsoft.WindowsAzure.Commands.Sync, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/Compute/Microsoft.WindowsAzure.Storage.DLL.
LOG: Assembly download was successful. Attempting setup of file: C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Compute\Microsoft.WindowsAzure.Storage.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: Where-ref bind Codebase does not match what is found in default context. Keep the result in LoadFrom context.
LOG: Binding succeeds. Returns assembly from C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Compute\Microsoft.WindowsAzure.Storage.dll.
LOG: Assembly is loaded in LoadFrom load context.

更新：使用 Nuget 包添加了所需的程序集。 仍然没有帮助。 想我不知何故需要告诉 Nuget 使其可用于 PowerShellISE，但没有看到参数......这必须接近解决方案。

PS C:\WINDOWS\system32> nuget install WindowsAzure.Storage -version 3.0.3
Attempting to resolve dependency 'Microsoft.Data.OData (ò 5.6.0)'.
Attempting to resolve dependency 'System.Spatial (= 5.6.0)'.
Attempting to resolve dependency 'Microsoft.Data.Edm (= 5.6.0)'.
Attempting to resolve dependency 'Newtonsoft.Json (ò 5.0.6)'.
Attempting to resolve dependency 'Microsoft.Data.Services.Client (ò 5.6.0)'.
Attempting to resolve dependency 'Microsoft.WindowsAzure.ConfigurationManager (ò 1.8.0.0)'.
Installing 'System.Spatial 5.6.0'.
Successfully installed 'System.Spatial 5.6.0'.
Installing 'Microsoft.Data.Edm 5.6.0'.
Successfully installed 'Microsoft.Data.Edm 5.6.0'.
Installing 'Microsoft.Data.OData 5.6.0'.
Successfully installed 'Microsoft.Data.OData 5.6.0'.
Installing 'Newtonsoft.Json 5.0.6'.
Successfully installed 'Newtonsoft.Json 5.0.6'.
Installing 'Microsoft.Data.Services.Client 5.6.0'.
Successfully installed 'Microsoft.Data.Services.Client 5.6.0'.
Installing 'Microsoft.WindowsAzure.ConfigurationManager 1.8.0.0'.
Successfully installed 'Microsoft.WindowsAzure.ConfigurationManager 1.8.0.0'.
Installing 'WindowsAzure.Storage 3.0.3.0'.
Successfully installed 'WindowsAzure.Storage 3.0.3.0'.

PS C:\WINDOWS\system32> F:\88acres\3-hive\hiveretrieval\error.ps1
WARNING: When submitting a query use the -RunAsFile switch to prevent errors with query lengths or special characters
Wait for the Hive job to complete ...


Cluster         : Crunch88
ExitCode        : 0
Name            : Hive: select * from sensor
PercentComplete : map 100% reduce 0%
Query           : select * from sensor_part where bld='B24' and tag='B24/AHU/LR/_1/OAT'
State           : Completed
StatusDirectory : /tagtransfer
SubmissionTime  : 04-Sep-15 08:03:50
JobId           : job_1440975278889_0021

Get-AzureHDInsightJobOutput : Could not load file or assembly 'Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, 
PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At F:\88acres\3-hive\hiveretrieval\error.ps1:18 char:1
+ Get-AzureHDInsightJobOutput -StandardError -JobId $job.JobId -Cluster ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-AzureHDInsightJobOutput], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.GetAzu 
   reHDInsightJobOutputCmdlet

Answer 1

我没有找到解决这个问题的好办法。 我在安装到 Visual Studio 2015 后遇到了同样的问题，所以如果有人有好的解决方案，我很乐意听到。 此时......我刚刚将 Microsoft.WindowsAzure.Storage.dll 版本 3.0.3.0 复制到我的 \\Program Files (x86)\\Microsoft SDKs\\Azure\\PowerShell\\ServiceManagement\\Azure\\HDInsight 文件夹（使首先备份较新版本）。 它现在工作正常，但这只是一个临时的解决方法。