从 .tfrecord 到 tf.data.Dataset 到 tf.keras.model.fit

Question

I am attemping to use Tensorflow (v2.0)'s Datasets API to pass large amounts of data to a tf.keras.model . Here is a simplified version of my dataset:

for rec in my_dataset:
    print(repr(rec))

$ {'feature0': <tf.Tensor: id=528, shape=(), dtype=float32, numpy=0.2963>,
'feature1': <tf.Tensor: id=618, shape=(), dtype=int64, numpy=0>,
'feature2': <tf.Tensor: id=620, shape=(), dtype=string, numpy=b'Inst1'>,
'target': <tf.Tensor: id=621, shape=(), dtype=int64, numpy=2>}
{'feature0': <tf.Tensor: id=528, shape=(), dtype=float32, numpy=0.4633>,
'feature1': <tf.Tensor: id=618, shape=(), dtype=int64, numpy=1>,
'feature2': <tf.Tensor: id=620, shape=(), dtype=string, numpy=b'Inst4'>,
'target': <tf.Tensor: id=621, shape=(), dtype=int64, numpy=0>}

...and so on. Each record in the my_dataset object is a dictionary with the features' (and target's) names as the keys and associated tensors as the values. I created the dataset from several .tfrecord files, so I'm constrained in the sense that each tensor corresponds to a tf.train.Example (wrapper) object. The dataset precisely matches the format seen in tensorflow documentation (see, for example, the last code example in https://www.tensorflow.org/tutorials/load_data/tfrecord#reading_a_tfrecord_file ).

I would like to use this dataset with keras. The tf.keras.model objects I'm working with all seem, for their fit function, to take as input a tuple representing the feature vector (X) and the target (y). I think I could figure out how to transform the tensors from my dataset into numpy arrays and pass them into the model that way, or iterate over the dataset using an iterator, but if I understand correctly that seems to defeat the whole purpose of using the Datasets API to begin with (see, for example, https://www.tensorflow.org/guide/keras/overview#train_from_tfdata_datasets ).

My question: what is the appropriate way to transform my_dataset into some form that tf.keras.model.fit() will receive? Or if this is the wrong question, what fundamental concepts am I missing that keep me from asking the right one? (For example, should the .tfrecord Examples be structured differently? Or, am I required to use an iterator instead of directly passing my_dataset to the model as I'd prefer?)

Answer 1

Unfortunately I was only able to find a workaround instead of an outright solution. Because tf.stack will only work on items of the same data type, I need to transform all data into floats during processing of the Examples (including one-hot encoding for all strings), and then use tf.stack on the resulting tensor:

def proces_example(serialized_example):
    feature_description = get_feature_desc()  # dictionary describing features and dtypes
    target_name = get_target_name()  # so we don't include the target in our feature vector
    parsed_example = tf.io.parse_single_example(serialized_example, feature_description)
    tensor_list = []
    for tensor in parsed_example:
        if tensor != target_name:
            parsed_example[tensor] = tf.dtypes.cast(parsed_example[tensor], tf.float32)
            tensor_list.append(parsed_example[tensor])
    X = tf.stack(tensor_list)
    y = parsed_example[target_name]
    return X, y