MirroredStrategy 導致 IndexError：使用 Keras 序列作為 model 輸入時從空列表中彈出

Question

雖然MirroredStrategy的IndexError: pop from empty list現在臭名昭著，並且有許多可能的原因，例如在以下問題中報告的：

• K.clear_session()引起的 MirroredStrategy IndexError
• AutoKeras 中的 MirroredStrategy IndexError
• 從檢查點訓練時出現 MirroredStrategy IndexError

等等，但都不適用於我的用例。

在我的用例中，我使用Keras Sequence對象來生成訓練輸入，因為我正在處理具有單個已知正 class 和未知負數的大型數據集（不適合 RAM）。

按照Keras 文檔和TensorFlow 文檔中可用的教程，我的代碼如下所示：

my_training_sequence = MySequenceObject()

if tf.config.list_physical_devices('GPU'):
    strategy = tf.distribute.MirroredStrategy(devices)
else:
    # Use the Default Strategy
    strategy = tf.distribute.get_strategy()

with strategy.scope():
    model = CreateMyKerasModel()
    # While in the TensorFlow documentation the compilation step
    # is shown OUTSIDE the scope, in the Keras one it happens
    # within the scope.
    # I  have found out that is NECESSARY to place it inside the scope
    # as the Keras Metrics need to be in the same strategy scope of the model
    # to work properly.
    model.compile(...)

# Then, OUSIDE from the score, run the fit
# which causes the IndexError
model.fit(my_training_sequence)

關於如何處理這個的任何想法？

Answer 1

在痛苦之后，我意識到在 Keras 文檔中，他們使用了TensorFlow 數據集對象。

現在，向量等正常輸入在擬合過程中轉換為數據集，因此不會導致問題，但目前 Keras 不支持將 Keras 序列自動轉換為數據集。 雖然我不知道這是為什么，但幸運的是，創建一個將序列轉換為數據集的方法相對容易。

不幸的是，它取決於您使用的 TensorFlow 的版本，因此在某些版本中您想要使用TensorSpec對象，而在較舊的版本中，只需將 tensorflow 數據類型和TensorShape組合即可。

在下面的示例中，我將展示一種高級方法來編寫可以轉換為數據集的 Keras 序列 class。 之后，我將鏈接到我已經以這種方式實現的所有 Keras 序列，作為后代的例子（或者我自己，一旦我忘記了這個惡魔般的東西的一些細節）。

import tensorflow as tf
import numpy as np
from packaging import version
from validate_version_code import validate_version_code


def tensorflow_version_is_higher_or_equal_than(tensorflow_version: str) -> bool:
    """Returns boolean if the TensorFlow version is higher than provided one.

    Parameters
    ----------------------
    tensorflow_version: str,
        The version of TensorFlow to check against.

    Raises
    ----------------------
    ValueError,
        If the provided version code is not a valid one.

    Returns
    ----------------------
    Boolean representing if installed TensorFlow version is higher than given one.
    """
    if not validate_version_code(tensorflow_version):
        raise ValueError(
            (
                "The provided TensorFlow version code `{}` "
                "is not a valid version code."
            ).format(tensorflow_version)
        )
    return version.parse(tf.__version__) >= version.parse(tensorflow_version)


class ExampleSequence:
    """Keras Sequence convertible into a TensorFlow Dataset."""

    def __init__(
        self,
        batch_size: int = 32,
        batches_per_epoch: int,
        # Your other parameters go here
    ):
        """

        Parameters
        --------------------------------
        batch_size: int = 32
            Size for the batches to generate,
            if the size is expected to be CONSTANT
            otherwise use None if some batches have different size
        batches_per_epoch: int
            The number of batches within an epoch
        """
        self._batch_size = batch_size
        self._batches_per_epoch = batches_per_epoch
        # Initialize the index of the batch for the Dataset calls
        self._current_index = 0
        # Your other parameters go here

    def __call__(self):
        """Return next batch using an infinite generator model."""
        self._current_index = (self._current_index + 1) % self._batches_per_epoch
        return self[self._current_index]

    def into_dataset(self) -> tf.data.Dataset:
        """Return dataset generated out of the current sequence instance.

        Implementative details
        ---------------------------------
        This method handles the conversion of this Keras Sequence into
        a TensorFlow dataset, also handling the proper dispatching according
        to what version of TensorFlow is installed in this system.

        Returns
        ----------------------------------
        Dataset to be used for the training of a model
        """

        #################################################################
        # Handling kernel creation when TensorFlow is a modern version. #
        #################################################################

        if tensorflow_version_is_higher_or_equal_than("2.5.0"):
            return tf.data.Dataset.from_generator(
                self,
                output_signature=(
                    (
                        tf.TensorSpec(
                            shape=(self._batch_size, 10),
                            dtype=tf.uint32
                        )
                    ),
                    tf.TensorSpec(
                        shape=(self._batch_size,),
                        dtype=tf.bool
                    )
                )
            )

        return tf.data.Dataset.from_generator(
            self,
            output_types=(
                (tf.uint32, ),
                tf.bool
            ),
            output_shapes=(
                (tf.TensorShape([self._batch_size, 10]),),
                tf.TensorShape([self._batch_size, ]),
            )
        )

    def __getitem__(self, idx: int):
        """Return batch corresponding to given index.

        Parameters
        ---------------
        idx: int,
            Index corresponding to batch to be returned.

        Returns
        ---------------
        Return Tuple containing X and Y numpy arrays corresponding to given batch index.
        """
        X = np.random.randint(shape=(self._batch_size, 10), dtype=np.uint32)
        y = np.random.randint(high=2, shape=(self._batch_size, ), dtype=np.bool)

        # Please do observe that the return type
        # has multiple layer of tuple wrapping, and they are ALL needed!
        # It is weird, but it is the only way this thing worked.
        return (((X, ), y,),)

然后，當你運行 fit 時，你可以使用：

model.fit(my_training_sequence.into_dataset())