[英]How to append data to a parquet file in Azure blob using Azure function
I am trying to append to a parquet file which is in Azure blob using Azure function c# script. 我正在尝试使用Azure函数c#脚本附加到Azure blob中的镶木地板文件。
I have been able to append to a locally created parquet file using Parquet.net package. 我已经能够使用Parquet.net包将其附加到本地创建的镶木文件中。 However, while I am trying to execute the code to append to a parquet file which is in Azure, I am getting errors.
但是,当我尝试执行代码以附加到Azure中的镶木地板文件时,出现了错误。
Below code works for a local parquet file append. 以下代码适用于本地镶木地板文件追加。
var ds = new DataSet(new DataField<int>("id"),new DataField<string>("city"));
ds.Add(1, "London");
using (Stream fileStream = File.Open(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
ParquetWriter.Write(ds, fileStream,CompressionMethod.None,null,null, true);
Console.Write("File writing completed successfully\n");
}
However the code below does not work for an Azure parquet file append 但是,下面的代码不适用于Azure Parquet文件追加
var ds = new DataSet(new DataField<int>("id"),new DataField<string>("city"));
ds.Add(1, "London");
Stream stream = new MemoryStream();
ParquetWriter.Write(ds, stream,CompressionMethod.None,null,null, false);
parquetBlob.AppendBlock(stream); //this line fails with error
I get following error: 我收到以下错误:
2018-07-05T05:07:14.479 [Info] Parquet file writing started
2018-07-05T05:07:14.667 [Info] Parquet file writing : successfully written to memory stream
2018-07-05T05:07:14.686 [Info] Exception while appending to parquet file: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 677
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 604
at Microsoft.WindowsAzure.Storage.Blob.CloudAppendBlob.AppendBlock(Stream blockData, String contentMD5, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudAppendBlob.cs:line 2145
at Submission#0.writeParquet(String data, CloudAppendBlob parquetBlob, TraceWriter log) in D:\home\site\wwwroot\EventHubTriggerCSharp2\run.csx:line 189
at Submission#0.WriteToBlob(String fileName, String data, TraceWriter log) in D:\home\site\wwwroot\EventHubTriggerCSharp2\run.csx:line 158
Request Information
RequestID:45299e54-001e-009d-7d1e-143e91000000
RequestDate:Thu, 05 Jul 2018 05:07:13 GMT
StatusMessage:The value for one of the HTTP headers is not in the correct format.
ErrorCode:InvalidHeaderValue
Any help will be highly appreciated. 任何帮助将不胜感激。
@Gaurav showed us the way. @Gaurav向我们展示了方法。
The value for one of the HTTP headers is not in the correct format.
If we check the RequestInformation, we will find Content-Length is 0. 如果检查RequestInformation,我们将发现Content-Length为0。
You need to seek stream back to the begin before appending it to the blob. 您需要先将流搜索回到开头,然后再将其附加到Blob。
Add stream.Position = 0
before parquetBlob.AppendBlock(stream);
在
parquetBlob.AppendBlock(stream);
之前添加stream.Position = 0
parquetBlob.AppendBlock(stream);
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.