[英]How do you Write an S3 Object using AWS Tools in powershell and verify using an md5checksum
I want to use the integrity features of AWS specified in: https://aws.amazon.com/premiumsupport/knowledge-center/data-integrity-s3/我想使用以下 AWS 的完整性功能: https://aws.amazon.com/premiumsupport/knowledge-center/data-integrity-s3/
to make md5checksums work.使 md5checksums 工作。 However, doing this:
但是,这样做:
Write-S3Object -BucketName BUCKETNAME -File .\test.txt -Key another_key4 -Metadata @{ md5checksum = 'asdfasdfasdfsa'} -HeaderCollection @{ md5checksum = 'asdfasdfasdfsa'} -Verbose
does not seem work as the operation completes successfully and uploads the file on S3.由于操作成功完成并在 S3 上上传文件,因此似乎不起作用。 I want to fail as the md5checksum is not 'asdfasdfasdfsa'
我想失败,因为 md5checksum 不是 'asdfasdfasdfsa'
When I look in the console, I can see that my -Metadata did at least something as:当我查看控制台时,我可以看到我的 -Metadata 至少做了一些事情:
How can I make so that the Write-S3Object operation fails unless I give it the correct md5checksum?除非我给它正确的 md5checksum,否则我怎样才能使 Write-S3Object 操作失败?
Below is based on https://forums.aws.amazon.com/message.jspa?messageID=414257 , pasting the entire (untested) code,so to avoid link riot下面是基于https://forums.aws.amazon.com/message.jspa?messageID=414257 ,粘贴整个(未经测试的)代码,以避免链接骚乱
#cleanup, zip, and push to s3
#The folders are organized to limit my need to upload big/long changes to s3
#Install is tiny so I can customize the install script quickly
#Config contains customization that is large, but stable. Skip if no changes.
#Package contains all the permanent zips and packages from vendors, very stable, but almost 1 GB so it takes 15 minutes to upload. I want to skip that upload if the file hasn't changed.
$magicSource = 'MagicSource'
$scriptPath = Split-Path -Parent $PSCommandPath # Get the location of this script.
$bucket = 'magic-xpi41-source-bucket'
if ((Test-S3Bucket -BucketName $bucket) -eq $false ){
Write-Output "Missing $bucket or you do not have access."
}
function Copy-ZipS3 {
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided'),
[string] $FileName = $(throw 'Error: A local filename must be provided')
)
if ( (Test-S3CRC -BucketName $BucketName -Key $Key -File $FileName) -eq 'ok' ){
Write-Output 'Skipping upload. File matches.'
return
}
try {
$timeS3 = Measure-Command {
Write-S3Object -BucketName $BucketName -Key $Key -File $FileName
}
Write-Output "Uploaded $filename to $bucketname -> $Key.`nTime to upload to S3: $timeS3"
}
catch {
$last_error = $Error[0]
"`nError/Exception:`nKey: $key`nFilename: $filename`n$last_error`n"
}
}
function Get-Md5Local {
param(
[string] $FileName = $(throw 'Error: A local filename must be provided')
)
$md5 = New-Object System.Security.Cryptography.MD5CryptoServiceProvider
$hash = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($FileName))).Replace('-', '').ToLower()
return $hash
}
function Get-Md5S3 {
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided')
)
$etag = (Get-S3Object -BucketName $BucketName -Key $Key)[0].ETag
$etag = $etag.Trim('"') # remove quotes from etag
return $etag
}
function Test-S3CRC {
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided'),
[string] $FileName = $(throw 'Error: A local filename must be provided'),
[switch] $Verbose = $false
)
$hash = Get-Md5Local -FileName $Filename
$etag = Get-Md5S3 -BucketName $BucketName -Key $Key
if ($Verbose) {"MD5 verified: $FileName ($hash) -> $BucketName : $Key ($etag)" }
if ($etag -ne $hash)
{
return "Tags are not equal:`nHash = $hash`nEtag = $etag"
}
else{
return 'ok'
}
}
function Repair-S3ETagOld{
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided')
)
if (Get-S3Object -BucketName $BucketName -Key $Key){
# to get the right etag, we must copy object back to itself
$message = Copy-S3Object -SourceBucket $BucketName -SourceKey $Key -DestinationKey "$Key.copy"
$message = Copy-S3Object -SourceBucket $BucketName -SourceKey "$Key.copy" -DestinationKey $Key
$message = Remove-S3Object -BucketName $BucketName -Key "$Key.copy" -Force
}
}
function Repair-S3ETag{
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided')
)
if (Get-S3Object -BucketName $BucketName -Key $Key){
# to get the right etag, we must copy object back to itself
$message = Copy-S3Object -SourceBucket $BucketName -SourceKey $Key -DestinationKey "$Key.copy"
$message = Remove-S3Object -BucketName $BucketName -Key $Key -Force
$message = Copy-S3Object -SourceBucket $BucketName -SourceKey "$Key.copy" -DestinationKey $Key
$message = Remove-S3Object -BucketName $BucketName -Key "$Key.copy" -Force
}
}
function Test-Zip {param ([String]$folder)
$installFolder = "$scriptPath\$folder"
if ((Test-Path -Path $installFolder\$magicSource ) -eq $false){
Write-Output "Missing $folder folder containing $folder files."
Write-Output "Run New-Item -ItemType directory -Path $installFolder\$magicSource"
exit
}
}
function New-Zip {param ([String]$folder)
$installFolder = "$scriptPath\$folder"
$timeZip = Measure-Command {
& "$env:ProgramFiles\7-Zip\7z.exe" u "$installFolder.zip" "$installFolder\*.*" -r
}
Write-Output "Time to zip $folder was: $timeZip"
}
function Copy-Source {param ([String]$folder)
Write-Output "Copying $folder to S3 if changed with Validation"
Test-Zip -folder $folder
New-Zip -folder $folder
Copy-ZipS3 -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip"
if ( (Test-S3CRC -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip") -ne 'ok' ){
Write-Output 'MD5 did not match. Running Repair.'
Repair-S3ETag -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip"
Test-S3CRC -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip" -Verbose
}
}
Copy-Source -folder Install
Copy-Source -folder Config
Copy-Source -folder Package
References:参考:
https://forums.aws.amazon.com/message.jspa?messageID=414257 https://forums.aws.amazon.com/message.jspa?messageID=414257
Write-S3Object -BucketName BUCKETNAME -File .\test.txt -Key another_key4 -Metadata @{ md5checksum = 'asdfasdfasdfsa'} -HeaderCollection @{ 'Content-Md5' = 'asdfasdfasdfsa'} -Verbose
Turn out you have to use the -HeaderCollection @{ 'Content-Md5' = 'asdfasdfasdfsa'}
instead of -HeaderCollection @{ 'md5checksum' = 'asdfasdfasdfsa'}
原来你必须使用
-HeaderCollection @{ 'Content-Md5' = 'asdfasdfasdfsa'}
而不是-HeaderCollection @{ 'md5checksum' = 'asdfasdfasdfsa'}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.