[英]HDF5 matrix append in python
例如,我們有矩陣(例如,我們要存儲numpy數組)並將其存儲在HDF5文件中,然后我們想通過在原始矩陣的末尾追加一些行來擴展矩陣(考慮到原始矩陣可能非常約數十Gb,無法加載到RAM中)
我們還希望能夠在不將整個矩陣加載到RAM中的情況下,從任意點(也許稱為slice(?))讀取矩陣中的幾行。
誰能提供一個示例如何在python中完成嗎?
更新:
我認為另一個選擇是numpy.memmap ,但似乎沒有附加。
這似乎也是一種選擇,但是它可以處理原始二進制數據,但是我想訪問矩陣。在這種情況下,我也不知道如何添加。
如果要使用HDF5文件,那么我建議使用一種可用的庫,例如Pytables。 我在這里發布並簡化了他們的教程: http : //pytables.github.io/usersguide/tutorials.html
from tables import *
# Define a user record to characterize some kind of particles
class Particle(IsDescription):
name = StringCol(16) # 16-character String
idnumber = Int64Col() # Signed 64-bit integer
ADCcount = UInt16Col() # Unsigned short integer
TDCcount = UInt8Col() # unsigned byte
grid_i = Int32Col() # integer
grid_j = Int32Col() # integer
pressure = Float32Col() # float (single-precision)
energy = FloatCol() # double (double-precision)
filename = "test.h5"
# Open a file in "w"rite mode
h5file = openFile(filename, mode = "w", title = "Test file")
# Create a new group under "/" (root)
group = h5file.createGroup("/", 'detector', 'Detector information')
# Create one table on it
table = h5file.createTable(group, 'readout', Particle, "Readout example")
# Fill the table with 10 particles
particle = table.row
for i in xrange(10):
particle['name'] = 'Particle: %6d' % (i)
particle['TDCcount'] = i % 256
particle['ADCcount'] = (i * 256) % (1 << 16)
particle['grid_i'] = i
particle['grid_j'] = 10 - i
particle['pressure'] = float(i*i)
particle['energy'] = float(particle['pressure'] ** 4)
particle['idnumber'] = i * (2 ** 34)
# Insert a new particle record
particle.append()
# Close (and flush) the file
h5file.close()
#now we will append some data to the table, after taking some slices
f=tables.openFile(filename, mode="a")
f.root.detector
f.root.detector.readout
f.root.detector.readout[1::3]
f.root.detector.readout.attrs.TITLE
ro = f.root.detector.readout
#generators work
[row['energy'] for row in ro.where('pressure > 10')]
#append some data
table = f.root.detector.readout
particle = table.row
for i in xrange(10, 15):
particle['name'] = 'Particle: %6d' % (i)
particle['TDCcount'] = i % 256
particle['ADCcount'] = (i * 256) % (1 << 16)
particle['grid_i'] = i
particle['grid_j'] = 10 - i
particle['pressure'] = float(i*i)
particle['energy'] = float(particle['pressure'] ** 4)
particle['idnumber'] = i * (2 ** 34)
particle.append()
table.flush()
f.close()
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.