Query Performance Issue in Linq

Question

I tried a query to fetch data from db but when I use add the following subqueries then it takes a lot of time, It fetches all the data from db then again uses subquery to fetch further data from memory.

I also tried to join with response table to fetch only required data but that still didn't solve the performance issue.

Here are the lines which causes performance issue:

projectType= internshipNameResponses.FirstOrDefault(y => y.QuestionID == 22).Answer,
additionalCriteria = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 27).Answer,
tamidStudent = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 28).Answer,
studentEmail = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 2134).Answer,
jobDesc = internshipNameResponses.Where(y => y.QuestionID == 17).Select(x => x.Answer).ToList(),
experience = internshipNameResponses.Where(y => y.QuestionID == 21).Select(x => x.Answer).ToList(),
VCF = internshipNameResponses.Where(y => y.QuestionID == 4239).Select(x => x.AnswerCode).ToList(),
lang = internshipNameResponses.Where(y => y.QuestionID == 23).Select(x => x.Answer).ToList(),
codingLang = internshipNameResponses.Where(y => y.QuestionID == 24).Select(x => x.Answer).ToList(),
academic = internshipNameResponses.Where(y => y.QuestionID == 25).Select(x => x.Answer).ToList(),
HPD = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 26).Answer,

Here is the complete Query:

var result = (from cr in CompanyRepresentatives.Where(x=> companyIds.Contains(x.ID) )
    join p in QuestionnaireResponses on cr.ID equals p.RespondentID
    join q in Responses on p.ID equals q.QuestionnaireResponseID into internshipNameResponses
    join x in Responses on p.ID equals x.QuestionnaireResponseID into posibleAnswerResponses
    from internshipNameRes in internshipNameResponses
    from possibleAnsRes in posibleAnswerResponses
    where internshipNameRes.QuestionID == 2130 &&
        possibleAnsRes.QuestionID == 29 &&
        p.QuestionnaireID == 2 &&
          
        cr.Active == true &&

          //Applying the name filter
          (string.IsNullOrEmpty(companyOrInternshipName) || cr.CompanyName.ToLower().Contains(companyOrInternshipName) ||
          (internshipNameRes != null && internshipNameRes.Answer.ToLower().Contains(companyOrInternshipName))) &&

          //Applying the status filter
          

          //Applying the year filter
          
          //applying the active filter
          (includeHidden || !p.IsHidden)
    orderby cr.CompanyName
    select new 
    {
        ID = p.ID,
        Name = internshipNameRes.Answer,
        CompanyID = cr.ID,
        Company = cr.CompanyName,
        CompanyEmail = cr.Email,
        CompanyDesc = cr.CompanyDescription,
        usOfficeCity = cr.USOffice_City,
        isrOfficeoth = cr.IsraelOffice_City_Other,
        CompanyRank = cr.Rank,
        projectType= internshipNameResponses.FirstOrDefault(y => y.QuestionID == 22).Answer,
        additionalCriteria = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 27).Answer,
        tamidStudent = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 28).Answer,
        studentEmail = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 2134).Answer,
        AvailablePositions = possibleAnsRes.Answer,
        FilledPositions = p.FilledVacancies,
        Status = p.Status,
        Visible = !p.IsHidden,
        DatePosted = p.Created,
        Rejected = p.IsInternshipRejected,
        jobDesc = internshipNameResponses.Where(y => y.QuestionID == 17).Select(x => x.Answer).ToList(),
        experience = internshipNameResponses.Where(y => y.QuestionID == 21).Select(x => x.Answer).ToList(),
        VCF = internshipNameResponses.Where(y => y.QuestionID == 4239).Select(x => x.AnswerCode).ToList(),
        lang = internshipNameResponses.Where(y => y.QuestionID == 23).Select(x => x.Answer).ToList(),
        codingLang = internshipNameResponses.Where(y => y.QuestionID == 24).Select(x => x.Answer).ToList(),
        academic = internshipNameResponses.Where(y => y.QuestionID == 25).Select(x => x.Answer).ToList(),
        HPD = internshipNameResponses.FirstOrDefault(y => y.QuestionID == 26).Answer,
        industry = cr.CompanyIndustry,
        companySize = cr.CompanySize,
        usOffice = cr.USOffice_City,
        isrOffice = cr.IsraelOffice_City,
        companyType = cr.CompanyType,
        market = cr.CompanyTargetMarket,
        financingStage = cr.FinancingStage
    }).ToList();

Answer 1

First of all, I would suggest not using FirstOrDefault (which returns null in this case, if nothing matches), as you are not handling the null value here. Instead use the function First , which will give you a better error description.

You could step through the code and see exactly, which line is causing the problem (it tells you the time after each line). As you go through the table internshipNameResponses very often, I would suggest going over it only once and filtering it, then use the filtered table:

var questionIDs = new []{17, 21, 22, 23, 24, 25, 26, 27, 28, 2134, 4239};
var filteredResponses = internshipNameResponses.Where(y => questionIDs.Contains(y.QuestionID)).ToList();

projectType = filteredResponses.FirstOrDefault(y => y.QuestionID == 22).Answer;
additionalCriteria = filteredResponses.FirstOrDefault(y => y.QuestionID == 27).Answer;
tamidStudent = filteredResponses.FirstOrDefault(y => y.QuestionID == 28).Answer;
studentEmail = filteredResponses.FirstOrDefault(y => y.QuestionID == 2134).Answer;
jobDesc = filteredResponses.Where(y => y.QuestionID == 17).Select(x => x.Answer).ToList();
experience = filteredResponses.Where(y => y.QuestionID == 21).Select(x => x.Answer).ToList();
VCF = filteredResponses.Where(y => y.QuestionID == 4239).Select(x => x.AnswerCode).ToList();
lang = filteredResponses.Where(y => y.QuestionID == 23).Select(x => x.Answer).ToList();
codingLang = filteredResponses.Where(y => y.QuestionID == 24).Select(x => x.Answer).ToList();
academic = filteredResponses.Where(y => y.QuestionID == 25).Select(x => x.Answer).ToList();
HPD = filteredResponses.FirstOrDefault(y => y.QuestionID == 26).Answer;

You will see that although more data is fetched this way from the table, doing it all at once instead of in 10 separate queries saves much time.

Answer 2

For a quick reference...

.NET 7 Amazing LINQ performance:

https://www.linkedin.com/posts/thekaranpargaien_net-benchmarks-improvements-activity-6977685759197634560-Vtck?utm_source=share&utm_medium=member_desktop