如何使用pymongo在na数组中搜索值

Question

I am coding a Web Crawler, now it is already working and I want to make a get_inverted_index function. Therefore, I have two collections: lexicon and documents. Inside each document of the documents lexicon, I have an array named words, which has the id and the font size of each word in each document (page). My next step would be iterate over the words and look for the documents who has each specific word, but I cannot see how to write the query for this request. I have tried the following code snippet:

k = {}
for word in self.lexicon.find():
    s = set()
    for page in self.documents.find({'words' : {'$in' : word['_id'}}):

But this query did not work properly. As an example, one entry from my lexicon collection:

{
    "_id": {
        "$oid": "54723c55b59c44a167ed3424"
    },
    "word": "google"
}

And an example from my documents collection:

{
    "_id": {
        "$oid": "54723c54b59c44a167ed3423"
    },
    "url": "http://www.google.com",
    "words": [
        [
            {
                "$oid": "54723c55b59c44a167ed3424"
            },
            7
        ],
        [
            {
                "$oid": "54723c55b59c44a167ed3425"
            },
            2
        ],
        [
            {
                "$oid": "54723c55b59c44a167ed3428"
            },
            0
        ],
        [
            {
                "$oid": "54723c55b59c44a167ed342b"
            },
            0
        ],
        [
            {
                "$oid": "54723c56b59c44a167ed342e"
            },
            0
        ],
        [
            {
                "$oid": "54723c5eb59c44a167ed3477"
            },
            0
        ]
    ]
}

@Edit

I have tried with regex as well, but with no success: (For testing the expression)

for page in documents.find({'words' : [ObjectId('547244abb59c44a167ed4a84'), {"$regex": "*"}]}):
    print page

Also

for page in documents.find({'words' : [{'$in' : ObjectId('547244abb59c44a167ed4a84')}, {'$regex': '*'}]}):

    print page

Answer 1

That is a really unfortunate choice of schema for the documents collection.

You say that you have an array named words which has the id and the font size of each word in each document. Unfortunately, you have this id and font size as another array. What would make sense would be to have the id and font size as named fields in a subdocument. To put it in more Pythonic terms, you want a list of dictionaries, not a list of lists.

{  "_id":   <id here>,
   "url": "http://www.google.com",
   "words": [
       { "id":<id>, "fs":7 },
       { "id":<id>, "fs":2 }
   ]
}

This will make it simple to query via documents.find({"words.id":<id>}) query. In addition, if you happen to want to track other things about each word, it won't be a mystery what that second number means.

While you can contrive to make a query which happens to return what you want for the schema you have, it's really not a very good fit to what it's describing. However, if you are determined to stay with your current structure, the proper way to query it would be

documents.find({'words':{'$elemMatch':{'0':word['_id']}}})

rather than using double $elemMatch, this syntax specifically looks for array element whose first element matches the _id in question.

Answer 2

looks like you need to search the documents collection on a level deeper.

As of now you search for the element

{
    "$oid": "54723c55b59c44a167ed3424"
}

And the $in operator of your documents collection compares it to the list elements such as:

[
    {
        "$oid": "54723c55b59c44a167ed3424"
    },
    7
]

which obviously aren't the same ever. Unfortunately I don't have a mongodb to test any but maybe that tip helps you a little to improve your query.

EDIT: Found an older question here regarding a similar problem maybe that'll helps. According to that post something like the following works:

for page in documents.find({'words':{$elemMatch:{$elemMatch:{$in:[word['_id']]}}}})