如何將 numpy 數組列表輸入到 TensorFlow 模型中？

Question

我有一個大的 numpy 數組列表，我想將它們輸入到 TensorFlow 模型中。 由於 RAM 內存問題，我無法將列表連接成一個。 下面，我使用以下代碼重新創建了我擁有的數據集：

train_data_list = []
number_of_patients = 20

for i in range(number_of_patients):
    sample_size = int(np.random.randint(low=2000, high=30000, size=1))
    sequence_length = 1024 # subsequence length
    feature_size = 3 # number of features e.g. vital sign 1, vital sign 2, vital sign 3 
    
    random_data = np.random.rand(sample_size, sequence_length, feature_size)
    train_data_list.append(random_data)

從這里你會得到一個 numpy 數組的列表，每個數組都屬於一個病人。 我已經設置了我的 TensorFlow 模型並希望將這些數據輸入 - TensorFlow 不接受列表，我無法將我的數據連接到一個單一的 numpy 數組中。

Answer 1

使用h5py很容易，我使用一些時間正確並修復了所有這些錯誤消息。

[ 樣本 ]：

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
DataSet
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()

# Create hdf5 file
hdf5_file = h5py.File(database_buffer, mode='w')

# Train images
hdf5_file['x_train'] = train_images
hdf5_file['y_train'] = train_labels

# Test images
hdf5_file['x_test'] = test_images
hdf5_file['y_test'] = test_labels

hdf5_file.close()

# Visualize dataset train sample
hdf5_file = h5py.File(database_buffer,  mode='r')

# Load features
x_train = hdf5_file['x_train'][0: 50000]
x_test = hdf5_file['x_test'][0: 10000]
y_train = hdf5_file['y_train'][0: 50000]
y_test = hdf5_file['y_test'][0: 10000]

# random pickup
index = random.randint(0, 50000)
image = hdf5_file['x_train'][index]
label = hdf5_file['y_train'][index]
plt.imshow(image)
plt.show()
print(label)

[ 輸出 ]：

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
None
cuda_malloc_async
[5]
2022-04-05 07:01:09.480563: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-04-05 07:01:10.068919: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:214] Using CUDA malloc Async allocator for GPU: 0
2022-04-05 07:01:10.069200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4634 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1
Epoch 1/10
2022-04-05 07:01:13.067862: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8100
1563/1563 [==============================] - 27s 12ms/step - loss: 0.7749 - accuracy: 0.1023
Epoch 2/10
1563/1563 [==============================] - 18s 12ms/step - loss: 0.6487 - accuracy: 0.1027
Epoch 3/10
1563/1563 [==============================] - 18s 12ms/step - loss: 0.6039 - accuracy: 0.1017
Epoch 4/10
1563/1563 [==============================] - 19s 12ms/step - loss: 0.3459 - accuracy: 0.0975
Epoch 5/10
1563/1563 [==============================] - 18s 12ms/step - loss: 0.3192 - accuracy: 0.0946
Epoch 6/10
1563/1563 [==============================] - 18s 11ms/step - loss: 0.2996 - accuracy: 0.0933
Epoch 7/10
1285/1563 [=======================>......] - ETA: 3s - loss: 0.2811 - accuracy: 0.0863