类型错误：“列表”对象不是迭代器 - Tensorflow 自定义指标回调

Question

尝试使用自定义指标回调与 Tensorflow 一起使用时遇到问题。 我在下面创建了一个最小的工作示例来帮助排除故障。 我在跑：

Windows 10
Python 3.6

scikit-learn==0.23.2
pandas==0.25.3
numpy==1.18.5
tensorflow==2.3.0

使用乳腺癌二进制数据集，我试图调用此处显示为解决方案的自定义指标，但遇到上述错误，可能是因为我没有正确使用它。

这段代码...

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score, f1_score
import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import Callback

# Get binary classification dataset
data = load_breast_cancer(as_frame=True)
print(data)
df = data['data']
df['target'] = data['target']

# Train Test split
train, test = train_test_split(data, test_size = 0.10, shuffle=False)

# Define features and labels
x_train = train.iloc[:, :-1]
y_train = train.iloc[:, -1]
x_test = test.iloc[:, :-1]
y_test = test.iloc[:, -1]

# https://github.com/keras-team/keras/issues/10472#issuecomment-472543538
class Metrics(Callback):
    
    def __init__(self, val_data, batch_size=20):
        super().__init__()
        self.validation_data = val_data
        self.batch_size = batch_size
    
    def on_train_begin(self, logs={}):
        # print(self.validation_data)
        self.val_f1s = []
        self.val_recalls = []
        self.val_precisions = []
        
    def on_epoch_end(self, epoch, logs={}):
        batches = len(self.validation_data)
        total = batches * self.batch_size
        
        val_pred = np.zeros((total,1))
        val_true = np.zeros((total))
        
        for batch in range(batches):
            xVal, yVal = next(self.validation_data)
            val_pred[batch * self.batch_size : (batch+1) * self.batch_size] = np.asarray(self.model.predict(xVal)).round()
            val_true[batch * self.batch_size : (batch+1) * self.batch_size] = yVal
            
        val_pred = np.squeeze(val_pred)
        _val_f1 = f1_score(val_true, val_pred)
        _val_precision = precision_score(val_true, val_pred)
        _val_recall = recall_score(val_true, val_pred)
        
        self.val_f1s.append(_val_f1)
        self.val_recalls.append(_val_recall)
        self.val_precisions.append(_val_precision)
        
        return

# Define a function that creates a basic model
def make_deep_learning_classifier():
    model = Sequential()
    model.add(Dense(64, activation='relu', input_dim=x_train.shape[1], kernel_initializer='normal'))
    model.add(Dense(32, activation='relu', input_dim=x_train.shape[1], kernel_initializer='normal'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])
    return model

# Get our model
model = make_deep_learning_classifier()
print(model.summary())

# Define some params
batch_size = 32

# Call our custom callback
callback = [Metrics(val_data=[x_test, y_test], batch_size=batch_size)] # < Issue here?

# Start training
model.fit(x_train, y_train, epochs=1000, batch_size=batch_size, verbose=1, callbacks=callback, validation_data=(x_test, y_test))
print(Metrics.val_precisions) # < Issue here?

...产生此回溯...

  File "test.py", line 91, in <module>
    model.fit(x_train, y_train, epochs=1000, batch_size=batch_size, verbose=1, callbacks=callback, validation_data=(x_test, y_test))
  File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1137, in fit
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\callbacks.py", line 416, in on_epoch_end
    callback.on_epoch_end(epoch, numpy_logs)
  File "test.py", line 54, in on_epoch_end
    xVal, yVal = next(self.validation_data)
TypeError: 'list' object is not an iterator

当我在callback变量val_data=[x_test, y_test]更改为val_data=(x_test, y_test)时，我得到...

TypeError: 'tuple' object is not an iterator

提出此回调解决方案的用户提到了一些有关生成器的内容，但我不确定它们是如何工作的。 只是想为 Tensorflow/Keras 定义我自己的自定义指标。 我不会使用这个确切的回调，但是一旦我运行了这个回调，我就可以将它修改为我自己的。 仅提供它作为似乎在 GitHub 帖子中起作用的示例，我希望有人能够指出我做错了什么。

谢谢！

更新

使用下面的解决方案，我尝试通过使用正确调用我的 val_data 上的迭代器函数

iter_val_data = iter(self.validation_data)
for batch in range(batches):
    xVal, yVal = next(iter_val_data)

但是后来我得到了太多的值来解包错误，所以我将其更改为：

iter_val_data = iter(self.validation_data)
for batch in range(batches):
    xVal = next(iter_val_data)
    yVal = next(iter_val_data)

然后我得到错误：

Traceback (most recent call last):
  File "test.py", line 89, in <module>
    model.fit(x_train, y_train, epochs=1000, batch_size=batch_size, verbose=1, callbacks=callback, validation_data=(x_test, y_test))
  File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1137, in fit
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\callbacks.py", line 416, in on_epoch_end
    callback.on_epoch_end(epoch, numpy_logs)
  File "test.py", line 53, in on_epoch_end
    val_pred[batch * self.batch_size : (batch+1) * self.batch_size] = np.asarray(self.model.predict(xVal)).round()
ValueError: could not broadcast input array from shape (57,1) into shape (32,1)

来自这里的想法？ 如果可以，请尝试在与上述相同的环境中运行代码。 谢谢！

Answer 1

正如您在此处看到的以及错误消息所述，您需要将 next() 与迭代器一起使用。 你在列表上调用next() ， next()怎么知道，下一个是哪个元素呢？ 为此，您需要一个迭代器来保存该状态。 所以这应该可以解决您的问题：

iter_val_data = iter(self.validation_data)
for batch in range(batches):
    xVal, yVal = next(iter_val_data)