MongoDB aggregation $lookup to a field that is an indexed array

Question

I am trying a fairly complex aggregate command on two collections involving $lookup pipeline. This normally works just fine on simple aggregation as long as index is set on foreignField .

But my $lookup is more complex as the indexed field is not just a normal Int64 field but actually an array of Int64 . When doing a simple find() , it is easy to verify using explain() that the index is being used. But explaining the aggregate pipeline does not explain whether index is being used in the $lookup pipeline. All my timing tests seem to indicate that the index is not being used. MongoDB version is 3.6.2. Db compatibility is set to 3.6.

As I said earlier, I am not using simple foreignField lookup but the 3.6-specific pipeline + $match + $expr ...

Could using pipeline be showstopper for the index? Does anyone have any deep experience with the new $lookup pipeline syntax and/or the index on an array field?

Examples

Either of the following works fine and if explained, shows that index on followers is being used.

db.col1.find({followers: {$eq : 823778}})
db.col1.find({followers: {$in : [823778]}})

But the following one does not seem to make use of the index on followers [there are more steps in the pipeline, stripped for readability].

db.col2.aggregate([
    {$match:{field: "123"}},
    {$lookup:{
       from: "col1",
       let : {follower : "$follower"},
       pipeline: [{
            $match: {
                $expr: {
                    $or: [
                        { $eq : ["$follower", "$$follower"] },                       
                        { $in : ["$$follower", "$followers"]}
                       ]
                }                        
            }
        }],
       as: "followers_all"
     }
}])

Answer 1

This is a missing feature which is going to part of 3.8 version.

Currently eq matches in lookup sub pipeline are optimised to use indexes. Refer jira fixed in 3.7.1 ( dev version).

Also, this may be relevant as well for non-multi key indexes.