如何索引 PyMongo 中已知字段的未知字段？

Question

I am trying to find unique words in millions of tweets and also I want to keep where each word appears. In addition to that, I am also grouping the words by their initial. Here is a sample code:

from pymongo import UpdateOne
# connect to db stuff
for word in words: # this is actually not the real loop I've used but it fits for this example
    # assume tweet_id's and position is calculated here
    initial = word[0]
    ret = {"tweet_id": tweet_id, "pos": (beg, end)} # additional information about word
    command = UpdateOne({"initial": initial}, {"$inc": {"count": 1}, "$push": {"words.%s" % word: ret}}, upsert=True)
    commands.append(command)
    if len(commands) % 1000 == 0:
        db.tweet_words.bulk_write(commands, ordered=False)
        commands = []

However, this is way slow to analyze all those tweets. I am guessing that my problem occurs because I don't use an index on words field.

Here is an sample output of a document:

{
    initial: "t"
    count: 3,
    words: {
        "the": [{"tweet_id": <some-tweet-id>, "pos": (2, 5)}, 
                {"tweet_id": <some-other-tweet-id>, "pos": (9, 12)}]
        "turkish": [{"tweet_id": <some-tweet-id>, "pos": (5, 11)}]
    }
}

I've tried to create indexes using the following codes (unsuccessfully):

db.tweet_words.create_index([("words.$**", pymongo.TEXT)])

or

db.tweet_words.create_index([("words", pymongo.HASHED)])

I've got errors like add index fails, too many indexes for twitter.tweet_words or key too large to index . Is there a way to do this with indexes? Or should change my approach the problem (maybe redesign the db)?

Answer 1

To be indexed, you need to keep your dynamic data in the values of the objects, not the keys. So I'd suggest you rework your schema to look like:

{
    initial: "t"
    count: 3,
    words: [
        {value: "the", tweets: [{"tweet_id": <some-tweet-id>, "pos": (2, 5)}, 
                                {"tweet_id": <some-other-tweet-id>, "pos": (9, 12)}]},
        {value: "turkish", tweets: [{"tweet_id": <some-tweet-id>, "pos": (5, 11)}]}
    ]
}

Which you could then index as:

db.tweet_words.create_index([("words.value", pymongo.TEXT)])