Keras 自定义损失 Function InvalidArgumentError: In[1] 不是矩阵。 相反，它具有形状 []

Question

我正在尝试使用 Spearman 等级相关系数来编写自定义损失 function。 我想计算每对 y_true 和 y_pred 样本之间的 Spearman 秩相关系数（每个样本是 8 个元素的数组；例如，[1 2 3 4 5 6 7 8] 和 [3 2 1 4 5 8 6 7] ）。

I have followed the indications of this answer ( How to compute Spearman correlation in Tensorflow ) and Keras documentation ( https://keras.io/api/losses/ ), however there must be something I'm skipping with regard to the output shape计算的损失。

使用此自定义 function 训练 model 会产生以下错误：

model.compile(loss=spearman_correlation, optimizer=tf.keras.optimizers.Adam())
model.fit(train_x, train_y,batch_size=64, epochs=2, validation_data=(test_x, test_y), callbacks=[model_checkpoint])

InvalidArgumentError:  In[1] is not a matrix. Instead it has shape []
     [[node gradient_tape/model_19/dense_19/MatMul_1 (defined at <ipython-input-46-7e6fc7cd1b39>:12) ]] [Op:__inference_train_function_300522]

我尝试了一种棘手的方法来解决这个问题，我使用了一个 Keras 损失 function 的工作示例，我只是用我的损失 ZC1C425268E68385D1AB5074C17A94F14 中计算的值来修改结果。 通过这种方式，训练 function 有效，但是，我认为这不是正确做事的方式，但我没有看到问题出在哪里。 Looking at the outputs of the prints in the custom function, can be seen that the shape and type of my loss output object and the tensorflow's loss function output object are the same.

这是我计算损失的方式：

def get_rank(y_pred):
    temp = sorted(y_pred, reverse=False)
    res = [temp.index(i) for i in y_pred]
    res = np.array(res)+1
    return(res)

def custom_spearman_correlation(y_true, y_pred):
    s_coefs = tf.map_fn(lambda k: 1-stats.spearmanr(k[0], get_rank(k[1]))[0], tf.stack([y_true, y_pred], 1), dtype=tf.float32)

    loss = s_coefs
    print("CUSTOM LOSS: ")
    print("Shape: " + str(loss.shape))
    print(type(loss))

    print("WORKING LOSS")
    squared_difference = tf.square(y_true - y_pred)
    w_loss = tf.reduce_mean(squared_difference, axis=-1)
    print("Shape: " + str(w_loss.shape))
    print(type(w_loss))

    print("TRICKY ANSWER: ")
    t_loss = w_loss*0 + loss
    print("Shape: " + str(t_loss.shape))
    print(type(t_loss))
    return loss
    #return w_loss
    #return t_loss

def spearman_correlation(y_true, y_pred):
    sp = tf.py_function(custom_spearman_correlation, [tf.cast(y_true, tf.float32), tf.cast(y_pred, tf.float32)], Tout = tf.float32)
    return (sp)

这是 output：

CUSTOM LOSS: 
Shape: (64,)
<class 'tensorflow.python.framework.ops.EagerTensor'>
WORKING LOSS
Shape: (64,)
<class 'tensorflow.python.framework.ops.EagerTensor'>
TRICKY ANSWER: 
Shape: (64,)

Answer 1

虽然我不确定，但我认为上述解决方案不允许正确更新 model 中不同参数的权重，因此我的 model 没有学习。 我一直在努力按照本网站的定义（ https://rpubs.com/aaronsc32/spearman-rank-correlation ）直接在 tensorflow 中实现斯皮尔曼等级相关系数，我已经达到了以下代码（我只是分享它以防有人发现它有用）。

@tf.function
def get_rank(y_pred):
  rank = tf.argsort(tf.argsort(y_pred, axis=-1, direction="ASCENDING"), axis=-1)+1 #+1 to get the rank starting in 1 instead of 0
  return rank

@tf.function
def sp_rank(x, y):
  cov = tfp.stats.covariance(x, y, sample_axis=0, event_axis=None)
  sd_x = tfp.stats.stddev(x, sample_axis=0, keepdims=False, name=None)
  sd_y = tfp.stats.stddev(y, sample_axis=0, keepdims=False, name=None)
  return 1-cov/(sd_x*sd_y) #1- because we want to minimize loss

@tf.function
def spearman_correlation(y_true, y_pred):
    #First we obtain the ranking of the predicted values
    y_pred_rank = tf.map_fn(lambda x: get_rank(x), y_pred, dtype=tf.float32)
    
    #Spearman rank correlation between each pair of samples:
    #Sample dim: (1, 8)
    #Batch of samples dim: (None, 8) None=batch_size=64
    #Output dim: (batch_size, ) = (64, )
    sp = tf.map_fn(lambda x: sp_rank(x[0],x[1]), (y_true, y_pred_rank), dtype=tf.float32)
    #Reduce to a single value
    loss = tf.reduce_mean(sp)
    return loss