如何将python中的字符串转换为数字向量，以便我可以将它们与其他向量进行比较？

Question

I am looking for a way to convert Strings into vectors of numbers in phyton. Like

"Hi how are you?" -> "29 73 281 38"
"How are you doing" -> "73 281 28 54"

I want to compare sentences from a user input to sentences out of a databese, which are stored as vectors.

Answer 1

I am assuming you are trying to create a dense vector representation for your input sentences.

See if below code helps.

sentences = ["Hi how are you?", "How are you doing"]

# Step 1: Create vocabulary - a set of distinct tokens from your input sentences 
vocab = set()
for sentence in sentences:
    tokens = sentence.split()
    for token in tokens:
        vocab.add(token)

# Step 2: Create a map (token: ID)
vocab_map = {}
for i, token in enumerate(sorted(vocab)): # sorted lexicographically for reproducibility 
    vocab_map[token] = i

# encode the sentences using the map you created in the previous step
for sentence in sentences:
    encoded_sentence = []
    tokens = sentence.split()
    for token in tokens:
        encoded_sentence.append(str(vocab_map[token]))
    print(' '.join(encoded_sentence))

Running the above code should get output the following:

0 4 2 6
1 2 5 3