"如何使用 tensorflow 标记文本？"

Question

我正在尝试使用以下代码对句子进行矢量化：

from tensorflow.keras.layers import TextVectorization

text_vectorization_layer =  TextVectorization(max_tokens=10000,
                                              ngrams=5,
                                              standardize='lower_and_strip_punctuation',
                                              output_mode='int',
                                              output_sequence_length = 15
                                              )

text_vectorization_layer(['BlackBerry Limited is a Canadian software'])

Answer 1

您必须首先使用adapt<\/code>方法或通过将词汇数组传递给层的vocabulary<\/code>参数来计算TextVectorization<\/code>层的词汇。 这是一个工作示例：

import tensorflow as tf

text_vectorization_layer =  tf.keras.layers.TextVectorization(max_tokens=10000,
                                              ngrams=5,
                                              standardize='lower_and_strip_punctuation',
                                              output_mode='int',
                                              output_sequence_length = 15
                                              )

text_vectorization_layer.adapt(['BlackBerry Limited is a Canadian software'])
print(text_vectorization_layer(['BlackBerry Limited is a Canadian software']))