Azure Blob存储和HDF文件存储

Question

I am in the middle of developing a cloud server and I need to store HDF files ( http://www.hdfgroup.org/HDF5/ ) using blob storage.

Functions related to creating, reading writing and modifying data elements within the file come from HDF APIs.

I need to get the file path to create the file or read or write it.

Can anyone please tell me how to create a custom file on Azure Blob ?

I need to be able to use the API like shown below, but passing the Azure storage path to the file. http://davis.lbl.gov/Manuals/HDF5-1.4.3/Tutor/examples/C/h5_crtfile.c

These files i am trying to create can get really huge ~10-20GB, So downloading them locally and modifying them is not an option for me.

Thanks

Shashi

Answer 1

One possible approach, admittedly fraught with challenges, would be to create the file in a temporary location using the code you included, and then use the Azure API to upload the file to Azure as a file input stream. I am in the process of researching how size restrictions are handled in Azure storage, so I can't say whether an entire 10-20GB file could be moved in a single upload operation, but since the Azure API reads from an input stream, you should be able to create a combination of operations that would result in the information you need residing in Azure storage.

Answer 2

Can anyone please tell me how to create a custom file on Azure Blob ?

I need to be able to use the API like shown below, but passing the Azure storage path to the file. http://davis.lbl.gov/Manuals/HDF5-1.4.3/Tutor/examples/C/h5_crtfile.c

Windows Azure Blob storage is a service for storing large amounts of unstructured data that can be accessed via HTTP or HTTPS . So from application point of view Azure Blob does not work as regular disk.

Microsoft provides quite good API (c#, Java) to work with the blob storage. They also provide Blob Service REST API to access blobs from any other language (where specific blob storage API is not provided like C++).

A single block blob can be up to 200GB so it should easily store files of ~10-20GB size.

I am afraid that the provided example will not work with Windows Azure Blob. However, I do not know HDF file storage; maybe they provide some Azure Blob storage support.