Any idea why this RavenDB query is returning only 6 results?

Question

We are stumped on this query in Raven and can not figure out what is wrong with it for the life of us. We have a pretty big index. It also has a transform that runs that filters out results based on a whitelist document that contains an array of tags.

Here is the index:

public class Entries_ByWhiteListSearchableFields
        : AbstractIndexCreationTask<Entry, Entries_ByWhiteListSearchableFields.Result>
    {
        public class Result
        {
            public string Id { get; set; }
            public IEnumerable<string> Search { get; set; }
            public DateTimeOffset? LastPublishedAtUtc { get; set; }
            public DateTimeOffset LastModifiedAtUtc { get; set; }
            public DateTimeOffset DisplayAtUtc { get; set; }
            public PublishStatus PublishStatus { get; set; }
            public IEnumerable<string> BylineIds { get; set; }
            public IEnumerable<string> TagIds { get; set; }
            public IEnumerable<string> ReferenceIds { get; set; }
            public string SourceStreamConfigId { get; set; }
            public bool Deleted { get; set; }
            public IEnumerable<string> FullyQualifiedTagIds { get; set; }
            public DateTimeOffset? ExpireAtUtc { get; set; }
        }

        public Entries_ByWhiteListSearchableFields()
        {
            this.Map = entries => from entry in entries
                                  let entrySection = entry.Tags.FirstOrDefault(tag => tag.Schema == "EntrySection")
                                  where entrySection != null
                                  select new
                                  {
                                      Id = entry.Id,
                                      Search = entry.Tags
                                          .Select(x => x.Label)
                                          .Concat(new[] { entry.Headline }),

                                      LastPublishedAtUtc = entry.LastPublishedAtUtc,
                                      LastModifiedAtUtc = entry.LastModifiedAtUtc,
                                      DisplayAtUtc = entry.LastPublishedAtUtc ?? entry.LastModifiedAtUtc,
                                      PublishStatus = entry.PublishStatus,
                                      BylineIds = entry.BylineIds,
                                      TagIds = entry.Tags.Select(x => x.Id),
                                      ReferenceIds = entry.References.Select(x => x.Id),
                                      Deleted = entry.Deleted,
                                      EntrySectionId = entrySection.Id,
                                      FullyQualifiedTagIds = entry.Tags.Select(t => t.Schema + "." + t.Id),
                                      ExpireAtUtc = entry.ExpireAtUtc
                                  };

            this.TransformResults =
                (database, entries) => from entry in entries
                                       let whitelist = database.Load<WhiteList>("whitelistdocid")
                                       where database.Load<Entry>(entry.Id)
                                                     .Tags
                                                     .Select(t => t.Schema + "." + t.Id)
                                                     .Intersect(
                                                        whitelist.WhiteList.Select(s => "EntrySection." + s))
                                                     .Any()
                                       select entry;
            this.Index(x => x.Search, FieldIndexing.Analyzed);
        }
    }

We have this exact same index already running, but without the transform, and it works perfectly.

This index also seems to work, but in one area, we have this query:

query = this.session.Advanced.LuceneQuery<Entry, Entries_ByWhiteListSearchableFields>()    
.Statistics(out stats)
.WhereEquals("Deleted", false)
.OrderBy("-DisplayAtUtc")
.Skip(queryModel.PageSize * (queryModel.Page - 1))
.Take(queryModel.PageSize)
.Include("BylineIds")
.Include("Tags,Id")
.Include("References,Id")
.Select(factory.GetEntryDashboardItemViewModel);

This query works perfect with out original index without the transform. However, with the new index, when we request pageSize of 100, it only returns 6 results. When we ask for 50, it returns only a few. When we ask for 25, it returns 0.

However, this line:

.OrderBy("-DisplayAtUtc")

Seems to be causing a problem. When we comment out this line, it works, but the results are not sorted obviously. When we put it back in, the same issue with only a few results being returned.

I renamed a few things to make more sense. Please let me know if more information is needed.

Any help or suggestions are appreciated.

Answer 1

I would assume that the where in the TransformResults . Note that TR happens after the paging has happened. So you feed it 100 items, the TR filter them out, but there is no attempt to get any additional results to fill the rest of the page.

In general, TR shouldn't be doing filtering.

You should be doing this in your query.