如何在加载另一个数据集时清理 GPU memory

问题描述

我在比较两种类型的输入数据（3 秒和 30 秒）的音频频谱图上训练 CNN.network。 这导致实验中不同的频谱图大小。

我正在使用它来获取数据：

def get_data(data_type, batch_size):
    assert data_type in ['3s', '30s'], "data_type shoulbe either 3s or 30s"
    if data_type == '3s':
        audio_dir = DATA_PATH / 'genres_3_seconds'
        max_signal_length_to_crop = 67_500
    elif data_type == '30s':
        audio_dir = DATA_PATH / 'genres_original'
        max_signal_length_to_crop = 660_000
    input_shape = (max_signal_length_to_crop, 1)

    train_ds, val_ds = tf.keras.utils.audio_dataset_from_directory(
        directory=audio_dir,
        batch_size=batch_size,
        validation_split=0.2,
        output_sequence_length=max_signal_length_to_crop,
        subset='both',
        label_mode='categorical'
    )
    test_ds = val_ds.shard(num_shards=2, index=0)
    val_ds = val_ds.shard(num_shards=2, index=1)
    return train_ds, val_ds, test_ds, input_shape

我正在使用这个 function 来创建模型。

def get_model(model_type, data_type, input_shape):
    if data_type == '3s':
        WIN_LENGTH = 1024 * 2
        FRAME_STEP = int(WIN_LENGTH / 4)  # / 4 a nie /2

    elif data_type == '30s':
        WIN_LENGTH = 1024 * 4
        FRAME_STEP = int(WIN_LENGTH / 2)  # / 4 a nie /2
    specrtogram_layer = 
    kapre.composed.get_melspectrogram_layer(input_shape=input_shape, win_length=WIN_LENGTH, hop_length=FRAME_STEP)
    model = Sequential([
        specrtogram_layer,
        *model_dict[model_type],
        Dense(units=10, activation='softmax', name='last_dense')
    ])
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=START_LR),
        loss=tf.keras.losses.CategoricalCrossentropy(),
        metrics=['accuracy'],
    )
    return model

model_dict = {
    'CNN_Basic': [
        Conv2D(filters=8, kernel_size=3, activation='relu'),
        MaxPooling2D(2),
        Conv2D(filters=16, kernel_size=3, activation='relu'),
        MaxPooling2D(2),
        Conv2D(filters=32, kernel_size=3, activation='relu'),
        MaxPooling2D(2),
        Flatten(),
        Dense(units=128, activation='relu'),
    ],
    ...
}

我在一个循环中对不同的架构进行了几个实验。 这是我的训练循环：

for data_type in ['3s', '30s']:
    train_ds, val_ds, test_ds, input_shape = get_data(data_type=data_type, batch_size=30)
    for model_type in ['CNN_Basic', ...]:
        model = get_model(model_type, input_shape=input_shape, data_type=data_type)
        model.fit(train_ds, epochs=epochs, validation_data=val_ds)

我得到的错误：

Traceback (most recent call last):
  File "...\lib\site-packages\tensorflow\python\trackable\base.py", line 205, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "...\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "...\lib\site-packages\tensorflow\python\framework\ops.py", line 1969, in _create_c_op
    raise ValueError(e.message)
ValueError: Exception encountered when calling layer "dense" (type Dense).

Dimensions must be equal, but are 17024 and 6272 for '{{node dense/MatMul}} = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false](Placeholder, dense/MatMul/ReadVariableOp)' with input shapes: [?,17024], [6272,128].

Call arguments received by layer "dense" (type Dense):
  • inputs=tf.Tensor(shape=(None, 17024), dtype=float32)

我认为这是由数据集引起的，因为我只有在 30 秒频谱图之后用 3 秒频谱图运行实验时才会出现此错误。 我每次都在创建新模型，并使用tf.keras.utils.audio_dataset_from_directory加载数据，并在以下循环迭代中将其加载到同一变量。