使用 tf.data 时如何对数据进行自定义预处理？

Question

I need some help help with tf.data.

I am doing a few experiments on SQUAD dataset. dataset structure given is like below:

row-1]  { conext: "some big string", question:"q string", "answer": "some ans" }

I would like to make use of tf.data for load and pre-processing . After loading, it is loaded in foll. format:

{
  context: Tensor("some big string"), 
  question:Tensor(q string),
  answer": Tensor(some ans) 
}

Now we want to pre-process the data. Now here pre-processing is not straightforward because values are Tensor objects.

Tensorflow provides some apis for such kind of pre-processing but what if I want to do my custom pre-processing or maybe I want to use spacy which just operates on raw datatypes like string and not tensors.

Basically I want help with this snippet:

def format_data(row):
  # Now I can access individual data row here. But value of row is in Tensor form.

  # Hence I can't use my custom function. How to use custom function or spacy function which operates on string and not on tensor?

  # I can use only below tf functions
  return tf.strings.regex_replace(row['context'],'some-regex',' ',True)


train = dataset.map(format_data).batch(2)
ist(train.take(1))

Answer 1

Following code worked:

def parse_str(str_tensor):
    raw_string = str_tensor.numpy().decode("utf-8") 

    # play with raw string
    raw_string = 'AAA'+raw_string     
    return raw_string

Call parse function:

def tf_pre_processing(row):
  return tf.py_function(parse_str, [row['context']], [tf.string])


train = t.map(tf_pre_processing).batch(1).take(1)

list(train)