Django - 从查询集中排除对象导致“表达式树太大”

Question

I'm working on a project that handles astronomical data (literally and figuratively) and have hit a roadblock when trying to exclude observations that require complex filtering.

To be more specific, each Observation has one or more SpectralWindow s. Users must be able to query observations by - among others - looking to their frequency coverage in relation to a band, a frequency range or an emission line. Regardless of the selected option, this coverage can be redshifted inside a specific z interval.

Since it's a bit hard to directly translate this to __lte -like selectors, I just filter the queryset and manually exclude observations that don't cover the calculated frequency range. Something like this:

for o in obs_result:
                covers = covers_ranges(z_bands, o) # <--- a more complex filtering solution
                if(not covers):
                    obs_result = obs_result.exclude(id=o.id) # <--- here's the problem!
                # update the min and max frequency values across the obs set
                else:
                    min_max = get_min_max_f(o)
                    min_freq = min(min_freq, min_max[0])
                    max_freq = max(max_freq, min_max[1])

The resulting query ends up like this:

SELECT ... FROM "common_observation" WHERE ("common_observation"."dec" >= 1.3365999999999998 AND "common_observation"."dec" <= 3.3366 AND "common_observation"."field_of_view" <= 300.0 AND "common_observation"."ra" >= 149.2375 AND "common_observation"."ra" <= 151.2375 AND NOT ("common_observation"."id" = 415) AND NOT ("common_observation"."id" = 416) AND NOT ("common_observation"."id" = 611) AND NOT ("common_observation"."id" = 612) AND NOT ("common_observation"."id" = 753) AND NOT ("common_observation"."id" = 758) AND NOT ("common_observation"."id" = 883) AND NOT ("common_observation"."id" = 884) AND NOT ("common_observation"."id" = 968) AND NOT ("common_observation"."id" = 984) AND NOT ("common_observation"."id" = 985) AND NOT ("common_observation"."id" = 995) AND NOT ("common_observation"."id" = 996) AND NOT ("common_observation"."id" = 997) AND NOT ("common_observation"."id" = 998) AND NOT ("common_observation"."id" = 999) AND NOT ("common_observation"."id" = 1000) AND NOT ("common_observation"."id" = 1001) AND NOT ("common_observation"."id" = 1002) AND NOT ("common_observation"."id" = 1052) AND NOT ("common_observation"."id" = 1053) AND NOT ("common_observation"."id" = 1086) AND NOT ("common_observation"."id" = 1087) AND NOT ("common_observation"."id" = 1088) AND NOT ("common_observation"."id" = 1089) AND NOT ("common_observation"."id" = 1090) AND NOT ("common_observation"."id" = 1091) AND NOT ("common_observation"."id" = 1092) AND NOT ("common_observation"."id" = 1093) AND NOT ("common_observation"."id" = 1099) AND NOT ("common_observation"."id" = 1100) AND NOT ("common_observation"."id" = 1111) AND NOT ("common_observation"."id" = 1154) AND NOT ("common_observation"."id" = 1156) AND NOT ("common_observation"."id" = 1157) AND NOT ("common_observation"."id" = 1158) AND NOT ("common_observation"."id" = 1159) AND NOT ("common_observation"."id" = 1204) AND NOT ("common_observation"."id" = 1205) AND NOT ...

...followed by a couple more hundreds of similar lines. When the list gets iterated over, this behemoth understandably throws an "expression tree is too large" error.

I already tried to just evaluate the queryset to a list and filter from there, but it won't include related objects, meaning I can't look into related objects/fields ( Observation.SpectralWindow.start , for instance).

What's the protocol for these problems, if there are any?

Answer 1

You may try to store ids in a list and then exclude all ids which are in the list, like this:

exclude_ids = []
for o in obs_result:
                covers = covers_ranges(z_bands, o) # <--- a more complex filtering solution
                if(not covers):
                    #obs_result = obs_result.exclude(id=o.id) # <--- here's the problem!
                    exclude_ids.append(o.id)
                # update the min and max frequency values across the obs set
                else:
                    min_max = get_min_max_f(o)
                    min_freq = min(min_freq, min_max[0])
                    max_freq = max(max_freq, min_max[1])
obs_result = obs_result.exclude(id__in = exclude_ids)