Pytorch 相当于 tensorflow keras StringLookup？

Question

. Answers to this question are eligible for a +150 reputation bounty. Jaime Ferrando Huertas wants to to this question:

I want to achieve similar behaviour with a pytorch layer as the one I mention I can do with StringLookup on tensorflow

I'm working with pytorch now, and I'm missing a layer: tf.keras.layers.StringLookup that helped with the processing of ids. Is there any workaround to do something similar with pytorch?

An example of the functionality I'm looking for:

vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d"], ["d", "a", "b"]])
layer = tf.keras.layers.StringLookup(vocabulary=vocab)
layer(data)

Outputs:
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[1, 3, 4],
       [4, 1, 2]])>

Answer 1

You can use Collections.Counter along with torchtext 's vocab object to construct a lookup function from your vocabulary. You can then easily pass sequences to this and get their encodings as a tensor:

from torchtext.vocab import vocab
from collections import Counter

tokens = ["a", "b", "c", "d"]
samples = [["a", "c", "d"], ["d", "a", "b"]]

# Build string lookup
lookup = vocab(Counter(tokens))

>>> torch.tensor([lookup(s) for s in samples])
tensor([[0, 2, 3],
        [3, 0, 1]])

Answer 2

you can use the library torchtext, just install it with python3 -m pip install torchtext

and you can is like this:

from torchtext.vocab import vocab
from collections import OrderedDict

tokens = ['a', 'b', 'c', 'd']
v1 = vocab(OrderedDict([(token, 1) for token in tokens]))
v1.lookup_indices(["a","b","c"])

and this is the result:

([0, 1, 2],)

Answer 3

Package torchnlp,

pip install pytorch-nlp

from torchnlp.encoders import LabelEncoder

data = ["a", "c", "d", "e", "d"]
encoder = LabelEncoder(data, reserved_labels=['unknown'], unknown_index=0)

enl = [encoder.encode(x) for x in data]

print(enl)

[tensor(1), tensor(2), tensor(3), tensor(4), tensor(3)]