将一个 tf.data.Datasets 与另一个的所有其他元素合并

Question

I would like to merge two tf.data.Dataset so that only every other sample of the first one is combined with the other, without any sample lost.

For example, let's have two lists of numbers:

ds1 = tf.data.Dataset.range(10)
ds10 = tf.data.Dataset.range(10, 60, 10)

I want to combine them so that samples from the second are added to the first, but only every other time:

0, 11, 2, 23, 4, 35, 6, 47, 8, 59

There is a zip method that enables to merge two datasets, but it does so by drawing a sample from each -- not combining samples would mean dropping a sample from ds10 , which is not what I want.

I could continue from there, and zipping ds10 with "dummy" samples that are dropped during the zip with ds1 , but it doesn't look very efficient.

Is there an efficient way to do that, without dropping samples (either real or "dummy")?

Answer 1

Try this:

def combine(pair,to_add):
    combined = [pair[0], pair[1] + to_add]
    return tf.data.Dataset.from_tensor_slices(combined)

ds1 = tf.data.Dataset.range(10)
ds2 = tf.data.Dataset.range(10,60,10)

combined = tf.data.Dataset.zip((ds1.batch(2),ds2)).flat_map(combine)

Explanation:

First, batch ds1.batch(2) .This produces [(0,1), (2,3), ...] .
Zip this to the other dataset to get [((0,1),10), ((2,3),20), ...] .
Undo the batching with flat_map and in the process combine every (a,b) with c in each [((a,b),c), ...] like [(a,b+c), ...] .
The result is then flattened to remove the braces and you get [0, 11, 2, 23, 4, 35, 6, 47, 8, 59] .
Batching and unbatching like this is a common pattern when dealing with several tf.data.Dataset s.