[Tensorflow 2]How to build data input pipeline for a multi-input multi-output model with data that has inconsistent shapes

Question

I'm using Tensorflow 2 and I need to build a multi-input multi-output model, and my data is timeseries data, which does not have a consistent shape for its time dimension. I've tried many ways but none worked due to the inconsistent shape.

There are three data and one of them is used twice. They have the format of (number of files, None, 5) , with the None dimension to be the inconsistent dimension.

Here's some testing codes that reproduce my issues, and I'm using a generator in this case, but feel free to change to whatever method. Could someone help me with this input pipeline?

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

dummy_1 = [[[1.1,2,3,4,5],[2,3,4,5,6],[3,4,5,6,7]],
           [[1.2,2,3,4,5],[2,3,4,5,6.8]],
           [[1.3,2,3,4,5],[2,3,4,5,6],[3,4,5,6,7],[4,5,6,7,8.9]]]

dummy_2 = [[[1.1,2,3,4,5],[2,3,4,5,6]],
           [[1.1,2,3,4,5],[2,3,4,5,6]],[3,4,5,6,7],
           [[1.3,2,3,4,5],[2,3,4,5,6]]]

dummy_3 = [[[1.5,2,3,4,5],[2,3,4,5,6]],
           [[1.6,2,3,4,5],[2,3,4,5,6]],[3,4,5,6,7],
           [[1.7,2,3,4,5],[2,3,4,5,6]]]

def gen():
    for i in range(len(dummy_1)):
        yield(dummy_1[i],dummy_2[i],dummy_2[i],dummy_3[i])

def custom_loss(y_true, y_pred):
    return tf.reduce_mean(tf.abs(y_pred - y_true))

class network():
    def __init__(self):
        input_1 = keras.Input(shape=(None,5))
        input_2 = keras.Input(shape=(None,5))
        output_1 = layers.Conv1DTranspose(16, 3, padding='same', activation='relu')(input_1)
        output_2 = layers.Conv1DTranspose(16, 3, padding='same', activation='relu')(input_2)
        
        self.model = keras.Model(inputs=[input_1, input_2],
                                 outputs=[output_1, output_2])
        
        # compile model
        self.model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.001),
                           loss={"mel_loss":custom_loss, "mag_loss":custom_loss})
    
    def train(self):
        self.dataset = tf.data.Dataset.from_generator(gen, 
                                                      (tf.float32, tf.float32, tf.float32, tf.float32))
        self.dataset.batch(32).repeat()
        
        self.model.fit(self.dataset,epochs=3)
        #self.model.fit([dummy_1, dummy_2],
        #               [dummy_2, dummy_3],
        #               epochs=3)

net = network()
net.train()

Answer 1

这对于 TF2 目前是不可能的，参考https://github.com/tensorflow/tensorflow/issues/45112