linq join with groupby and max issues

Question

I'm playing around with Entity Framework and having a lot of trouble replicating the linq query that is relatively straightforward in SQL.

public class ReportQueueServiceCommandDTO
{
    public int Id { get; set; }
    public DateTime CommandDatetime { get; set; }
    public int CommandId { get; set; }
    public bool IsCommandAcknowledged { get; set; }
    public int QueueNumber { get; set; }
}

public IQueryable<ReportQueueServiceCommandDTO> GetLastestUnprocessedRequestsPerQueue()
{

    // Get the maximum Acknowledged record per queue
    var maxDateTimesPerAcknowledgedQueue = this.Get()
        .Where(w => w.IsCommandAcknowledged)
        .GroupBy(gb => gb.QueueNumber)
        .Select(s => new ReportQueueServiceCommandDTO()
        {
            Id = (int)s.Max(m => m.Id),
            CommandDatetime = s.FirstOrDefault(fod => fod.Id == s.Max(max => max.Id)).CommandDatetime, 
            CommandId = s.FirstOrDefault(fod => fod.Id == s.Max(max => max.Id)).CommandId,
            IsCommandAcknowledged = true,
            QueueNumber = s.Key
        });

    // Get the maximum unacknowledged record which had an ID greater than the maximum acknowledged record per queue.
    // If the queue entry does not yet exist from MaxDateTimesPerAcknowledgedQueue, use the record anyway (left join-ish)
    return from record in _context.ReportQueueServiceCommands
           from maxRecord in MaxDateTimesPerAcknowledgedQueue.DefaultIfEmpty()
           where record.QueueNumber == (maxRecord == null ? record.QueueNumber : maxRecord.QueueNumber)
             && record.Id > (maxRecord == null ? 0 : maxRecord.Id)
             && !record.IsCommandAcknowledged
           group record by record.QueueNumber into groupedRecords
           select new ReportQueueServiceCommandDTO()
           {
               Id = (int)groupedRecords.Max(m => m.Id),
               CommandDatetime = groupedRecords.FirstOrDefault(fod => fod.Id == groupedRecords.Max(max => max.Id)).CommandDatetime,
               CommandId = groupedRecords.FirstOrDefault(fod => fod.Id == groupedRecords.Max(max => max.Id)).CommandId,
               IsCommandAcknowledged = false,
               QueueNumber = groupedRecords.Key
           };

}

Using the above code i get an exception

The argument to DbIsNullExpression must refer to a primitive, enumeration, or reference type

when attempting to iterate through the results of GetLatestUnprocessedRequestsPerQueue()

If I change my query to:

return from record in _context.ReportQueueServiceCommands
       from maxRecord in MaxDateTimesPerAcknowledgedQueue
       where record.QueueNumber == maxRecord.QueueNumber
         && record.Id > maxRecord.Id
         && !record.IsCommandAcknowledged
       group record by record.QueueNumber into groupedRecords
       select new ReportQueueServiceCommandDTO()
       {
           Id = (int)groupedRecords.Max(m => m.Id),
           CommandDatetime = groupedRecords.FirstOrDefault(fod => fod.Id == groupedRecords.Max(max => max.Id)).CommandDatetime,
           CommandId = groupedRecords.FirstOrDefault(fod => fod.Id == groupedRecords.Max(max => max.Id)).CommandId,
           IsCommandAcknowledged = false,
           QueueNumber = groupedRecords.Key
       };

I don't get my error, and I almost get the results I want, however if that queue has happened to not have a successfully processed command, then there will be no record for that queue in maxDateTimesPerAcknowledgedQueue .

EDIT: Given data:

    Id    CommandDatetime    CommandId    IsCommandAcknowledged    QueueNumber
    0    2014-01-01          1            0                        1 -- should be returned as it is the maximum unacknowledged record for this queue when there are *no* acknowledged records for the queue.
    1    2013-12-31          1            0                        2 -- should not be returned as it is not the maximum unacknowledged record greater than the maximum acknowledged record
    2    2014-01-01          1            1                        2 -- should not be returned as already acknowledged
    3    2014-01-01          1            0                        2 -- should not be returned as it is not the maximum unacknowledged record greater than the maximum acknowledged record
    4    2014-01-02          1            0                        2 -- should be returned as it is the maximum unackowledged record past its maximum acknowledged record
    5    2014-01-01          1            1                        3 -- should not be returned as there are no "unacknowledged" records for this queue

Expected Result:

    Id    CommandDatetime    CommandId    IsCommandAcknowledged    QueueNumber
    0    2014-01-01          1            0                        1
    4    2014-01-01          1            0                        2

Actual Result:

    Id    CommandDatetime    CommandId    IsCommandAcknowledged    QueueNumber
    4    2014-01-01          1            0                        2

The issue with the "Actual result" is there have not been any acknowledged records for Queue 1, so that record is not being returned (but i need it to be). Might be reiterating here, but what I'm trying to accomplish is:

Return the maximum unacknowledged record per queue, greater than the maximum acknowledged record per queue, if no acknowledged record exists for a queue, return the maxumum unacknowledged record.

I hope it makes sense what I'm trying to accomplish, can anyone help me out?

Answer 1

Check for null: maxRecord == null is the problem.

use ToList() before checking for null.

Entity Framework: The argument to DbIsNullExpression must refer to a primitive or reference type

Answer 2

I ended up finding a way to accomplish what I wanted, though it feels extremely clunky and I'm wondering if there is a better way to do it:

/// <summary>
/// Get the latest unprocessed batch service command per queue
/// </summary>
/// <returns></returns>
public IQueryable<ReportQueueServiceCommandDTO> GetLatestUnprocessedCommandFromQueue()
{

    // Get the maximum Acknowledged record per queue
    var maxDateTimePerAcknowledgedQueue = this.Get()
        .Where(w => w.IsCommandAcknowledged)
        .GroupBy(gb => gb.QueueNumber)
        .Select(s => new ReportQueueServiceCommandDTO()
        {
            Id = (int)s.Max(m => m.Id),
            CommandDatetime = s.FirstOrDefault(fod => fod.Id == s.Max(max => max.Id)).CommandDatetime,
            CommandId = s.FirstOrDefault(fod => fod.Id == s.Max(max => max.Id)).CommandId,
            IsCommandAcknowledged = true,
            QueueNumber = s.Key
        });

    // Get the maximum unacknowledged record per queue
    var maxDateTimePerUnacknowledgedQueue = this.Get()
        .Where(w => !w.IsCommandAcknowledged)
        .GroupBy(gb => gb.QueueNumber)
        .Select(s => new ReportQueueServiceCommandDTO()
        {
            Id = (int)s.Max(m => m.Id),
            CommandDatetime = s.FirstOrDefault(fod => fod.Id == s.Max(max => max.Id)).CommandDatetime,
            CommandId = s.FirstOrDefault(fod => fod.Id == s.Max(max => max.Id)).CommandId,
            IsCommandAcknowledged = false,
            QueueNumber = s.Key
        });

    // Get the maximum unacknowledged record which had an ID greater than the maximum acknowledged record per queue.
    // If the queue entry does not yet exist from MaxDateTimesPerAcknowledgedQueue, use the record anyway (left join-ish)
    return (from unack in maxDateTimePerUnacknowledgedQueue
           join ack in maxDateTimePerAcknowledgedQueue on unack.QueueNumber equals ack.QueueNumber into joinedRecords
           from subAck in joinedRecords.Where(w => w.Id > unack.Id).DefaultIfEmpty()
           select new ReportQueueServiceCommandDTO()
           {
               Id = (subAck.Id == null ? unack.Id : subAck.Id),
               CommandDatetime = (subAck.CommandDatetime == null ? unack.CommandDatetime : subAck.CommandDatetime),
               CommandId = (subAck.CommandId == null ? unack.CommandId : subAck.CommandId),
               IsCommandAcknowledged = (subAck.IsCommandAcknowledged == null ? unack.IsCommandAcknowledged : subAck.IsCommandAcknowledged),
               QueueNumber = (subAck.QueueNumber == null ? unack.QueueNumber : subAck.QueueNumber)
           })
           .Where(w => !w.IsCommandAcknowledged);
}