How to perform multiple projections and lookups as aggregate in MongoDB using BsonDocument C# .NET driver?

Question

I can't achieve to perform multiple projections and lookups using aggregate functionality with the C# .NET driver.

Here the query I need to replicate with mongodb C# .NET driver.

db.organizations.aggregate([
  {
    "$project": {
      "_id": {
        "$toString": "$_id"
      },
      "Name": "$Name",
      "LastDateUploaded": "$LastDateUploaded",
      "LastDateEvaluated": "$LastDateEvaluated",
      "PriorityId": "$PriorityId"
    }
  },
  {
    "$lookup": {
      "from": "workflows",
      "localField": "_id",
      "foreignField": "Data.OrganizationId",
      "as": "RelatedWorkflows"
    }
  },
  {
    "$lookup": {
      "from": "priorities",
      "localField": "PriorityId",
      "foreignField": "_id",
      "as": "Priority"
    }
  },
  {
    "$unwind": "$Priority"
  },
  {
    "$project": {
      "Name": "$Name",
      "WorkflowCounter": {
        "$size": "$RelatedWorkflows"
      },
      "LastDateUploaded": "$LastDateUploaded",
      "LastDateEvaluated": "$LastDateEvaluated",
      "Priority": "$Priority.Value"
    }
  },
  {
    "$sort": {
      "Priority": -1,
      "WorkflowCounter": 1,
      "LastDateUploaded": -1,
      "LastDateEvaluated": -1
    }
  }
])

I have tried to do something like this:

public class Organization
    {
        [BsonId]
        [BsonRepresentation(BsonType.ObjectId)]
        public string Id {get; set;}
        public string NPOId { get; set; }
        public string Name { get; set; }
        public string Address { get; set; }
        [BsonRepresentation(BsonType.ObjectId)]
        public string PriorityId { get; set; }
        [BsonIgnoreIfDefault]
        public Priority Priority { get; set; }
        public string City { get; set; }
        public string County { get; set; }
        [MaxLength(2)]
        public string State { get; set; }
        [BsonRepresentation(BsonType.ObjectId)]
        public string CountryId { get; set; }
        [BsonIgnoreIfDefault]
        public Country Country { get; set; }
        [MaxLength(5)]
        public string Zip { get; set; }
        public string TaxCode { get; set; }
        public string ParentTaxCode { get; set; }
        [BsonRepresentation(BsonType.ObjectId)]
        public string CharityTypeId { get; set; }
        [BsonRepresentation(BsonType.ObjectId)]
        public string PreferredExtractorId { get; set; }
        [BsonIgnoreIfDefault]
        public User PreferredExtractor{ get; set; }
        [BsonIgnoreIfDefault]
        public CharityType CharityType { get; set; }
        public string URL { get; set; }
        public decimal Revenue { get; set; }
        public decimal Expenses { get; set; }
        public decimal Assets { get; set; }
        public decimal Excess { get; set; }
        public string IRSType { get; set; }
        public string ICPAScore { get; set; }
        public string ICPBScore { get; set; }
        public string[] DefaultRules { get; set; }
        [BsonIgnore]
        public long NumberOfWorkflows { get; set; }
        public DateTime? LastDateUploaded { get; set; }
        public DateTime? LastDateEvaluated { get; set; }
    }

var organizations = db.GetCollection<Organization>("catalog.organizations");
            var aggregation = organizations.Aggregate().Match(_ => true);

            var projectionOne = new BsonDocument {
                { "$project", new BsonDocument { { "_id", new BsonDocument { { "$toString", "$_id" } } } } },
                { "Name", "$Name"},
                { "LastDateUploaded", "$LastDateUploaded" },
                { "LastDateEvaluated", "$LastDateEvaluated"},
                { "PriorityId", "$PriorityId"}
            };

            aggregation.Project(projectionOne);

            aggregation.AppendStage<Organizations>(new BsonDocument {
                { "$lookup", new BsonDocument { { "from", "wcf.workflows" }, { "localField", "_id" }, { "foreignField", "Data.OrganizationId" }, { "as", "RelatedWorkflows" } } }
            });

            aggregation.AppendStage<Organizations>(new BsonDocument {
                { "$lookup", new BsonDocument { { "from", "catalog.priorities" }, { "localField", "PriorityId" }, { "foreignField", "_id" }, { "as", "Priority" } } }
            });

            aggregation.AppendStage<Organizations>(new BsonDocument {
                { "$unwind", "$Priority" }
            });

            aggregation.AppendStage<Organizations>(new BsonDocument {
                { "$project", new BsonDocument { { "Name", "$Name" }, { "WorkflowCounter", new BsonDocument { { "$size", "$RelatedWorkflows" } } } } },
                { "LastDateUploaded", "$LastDateUploaded" },
                { "LastDateEvaluated", "$LastDateEvaluated"},
                { "PriorityValue", "$Priority.Value"}
            });

            aggregation.AppendStage<Organizations>(new BsonDocument {
                { "$sort", new BsonDocument { { "Priority", 1 }, { "WorkflowCounter", 1 }, { "LastDateUploaded", -1 }, { "LastDateEvaluated", -1} } }
            });

            var organizationsList = await aggregation.ToListAsync();

However This is not working, for example I inverted the sort options and always is returning the same value. I tried to get the string representation of the mongo query bu I got only

aggregate([{ "$match" : { } }])

and is not appending the aggregate definitions.

I tried to use fluent notation to perform the lookups but I need to perform the lookup using an ObjectId and string fields, so for that reason I'm parsing firstly on mongodb side the ObjectId to string with the first projection and seems like there is not equivalence on fluent notation to transform ObjectId into string in order to perform the lookup (aka join).

Here the testing proof that the query is working properly on mongo shell: https://mongoplayground.net/p/dKT8uQHjHnd

I expected to get the first document from the list generated like in the mongo playground example, but I always obtain the first element of the collection.

Answer 1

I ended doing the next:

        var organizations = db.GetCollection<Organization>("catalog.organizations");
        var projectionOne = new BsonDocument {
                                                { "$project", new BsonDocument { 
                                                        { "_id", new BsonDocument { { "$toString", "$_id" } } },
                                                        { "Name", "$Name"},
                                                        { "LastDateUploaded", "$LastDateUploaded" },
                                                        { "LastDateEvaluated", "$LastDateEvaluated"},
                                                        { "PriorityId", "$PriorityId"}
                                                    } 
                                                }
        };

        var lookupOne = new BsonDocument {
            { "$lookup", new BsonDocument { { "from", "wfc.workflows" }, { "localField", "_id" }, { "foreignField", "Data.OrganizationId" }, { "as", "RelatedWorkflows" } } }
        };
        var lookupTwo = new BsonDocument {
            { "$lookup", new BsonDocument { { "from", "catalog.priorities" }, { "localField", "PriorityId" }, { "foreignField", "_id" }, { "as", "Priority" } } }
        };
        var unwindTwo = new BsonDocument {
            { "$unwind", "$Priority" }
        };

        var projectionTwo = new BsonDocument {
                                            { "$project", new BsonDocument { 
                                                    { "Name", "$Name" }, 
                                                    { "WorkflowCounter", new BsonDocument { { "$size", "$RelatedWorkflows" } } },
                                                    { "LastDateUploaded", "$LastDateUploaded" },
                                                    { "LastDateEvaluated", "$LastDateEvaluated"},
                                                    { "PriorityValue", "$Priority.Value"}
                                                } 
                                            }
        };

        var sort = new BsonDocument {
            { "$sort", new BsonDocument { { "PriorityValue", -1 }, { "WorkflowCounter", 1 }, { "LastDateUploaded", -1 }, { "LastDateEvaluated", -1} } }
        };

        var aggregation = organizations.Aggregate<OrganizationWithWorkflows>(new [] {projectionOne, lookupOne, lookupTwo, unwindTwo, projectionTwo, sort});

        var organizationsList = await aggregation.ToListAsync();

And I created this class to get the last projection results:

 [BsonIgnoreExtraElements]
 public class OrganizationWithWorkflows
 {
      public string Id { get; set; }
      public string Name { get; set; }
      public long WorkflowCounter {get; set;}
      public DateTime? LastDateUploaded { get; set; }
      public DateTime? LastDateEvaluated { get; set; }
      public int PriorityValue { get; set; }
 }

It took me a while but I finally resolved my challenge here, which in part was performing a lookup using ObjectId and string fields applying a cast from ObjectId to string and within the same query applying multiple projections and multiple lookups.

Please notice that the field counter is the size of the elements contained in the resultant array of RelatedWorkflows , but this can be done as well applying an unwind after the lookupOne and grouping to get the counter value with expression { $sum ,1 } but on my case it was enough to use the $size expression from the mentioned array.

I hope this can help to all those who are having little troubles understanding how to work with MongoDB Aggregation and C# .NET Driver using BsonDocuments as this query was not possible to be created using fluent notation because of the conversion of the ObjectId into string and use that one for lookupOne, maybe I'm wrong with this last part, so if you know how to do it, believe I will be glad to learn it as well.

In order to get my solution I found this awesome post from Mikael Koskinen and reading the mongodb documentation to use expression $toString