Queries for ModelMultipleChoiceFilter executed twice

[valentijnscholten]

Hi,

I am seeing that for every field that is a foreign key, that every query used to populate the dropdown list is executed twice.

I have stripped down my big filter down to the simplified filter below, and I am still seeing 2 identical queries for prod_type.

Models:

class Product(models.Model):
    ...
	prod_type = models.ForeignKey(Product_Type, related_name='prod_type',
                                  null=True, blank=True, on_delete=models.CASCADE)
	...

class Product_Type(models.Model):
	...
    name = models.CharField(max_length=255, unique=True)
    updated = models.DateTimeField(auto_now=True, null=True)
    created = models.DateTimeField(auto_now_add=True, null=True)

    class Meta:
        ordering = ('name',)

    def __unicode__(self):
        return self.name

    def __str__(self):
        return self.name

Filter:

class ProductFilter2(DojoFilter):
    prod_type = ModelMultipleChoiceFilter(
        queryset=Product_Type.objects.all().order_by('name'),
        label="Product Type")

    def __init__(self, *args, **kwargs):
        self.user = None
        if 'user' in kwargs:
            self.user = kwargs.pop('user')

        super(ProductFilter2, self).__init__(*args, **kwargs)

In a function based view:

filter = ProductFilter2(request.GET, queryset=prods, user=request.user)

Template:

<div class="filter-set">
    <form method="get">
        {{ filter.form.as_p }}
        <input type="submit" />
    </form>
</div>

Queries observed with django debug toolbar:

SELECT `dojo_product_type`.`id`,
       `dojo_product_type`.`name`,
       `dojo_product_type`.`updated`,
       `dojo_product_type`.`created`
  FROM `dojo_product_type`
 ORDER BY `dojo_product_type`.`name` ASC

The above are just examples, I have at least 5 other models and as many foreign key fields and all seem to trigger two queries per field.

It could be something I am overlooking, but I can't almost make my example any simpler. When I remove the prod_type field or remove the form.as_p from the template, both queries dissappear.

It happens both with autogenerated filters when using Meta model + fields, but also with a manual filter like the above.

requirements.txt:

django-filter==2.2.0
Django==2.2.12

Both queries have an identical stacktrace/traceback:

/home/user/venv_dd/lib/python3.6/site-packages/django/contrib/staticfiles/handlers.py in __call__(65)
  return self.application(environ, start_response)
/c/Data/dd/dd/middleware.py in __call__(62)
  response = self.get_response(request)
/c/Data/dd/dd/middleware.py in __call__(47)
  response = self.get_response(request)
/c/Data/dd/dd/product/views.py in product(79)
  'user': request.user})
/home/user/venv_dd/lib/python3.6/site-packages/django/shortcuts.py in render(36)
  content = loader.render_to_string(template_name, context, request, using=using)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader.py in render_to_string(62)
  return template.render(context, request)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/backends/django.py in render(61)
  return self.template.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(171)
  return self._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(150)
  return compiled_parent._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(150)
  return compiled_parent._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(62)
  result = block.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(188)
  return template.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(173)
  return self._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(987)
  output = self.filter_expression.resolve(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in resolve(671)
  obj = self.var.resolve(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in resolve(796)
  value = self._resolve_lookup(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in _resolve_lookup(858)
  current = current()
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/forms.py in as_p(304)
  errors_on_separate_row=True,
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/forms.py in _html_output(243)
  'field_name': bf.html_name,
/home/user/venv_dd/lib/python3.6/site-packages/django/utils/html.py in <lambda>(388)
  klass.__str__ = lambda self: mark_safe(klass_str(self))
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/boundfield.py in __str__(33)
  return self.as_widget()
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/boundfield.py in as_widget(93)
  renderer=self.form.renderer,
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in render(241)
  context = self.get_context(name, value, attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in get_context(678)
  context = super().get_context(name, value, attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in get_context(639)
  context['widget']['optgroups'] = self.optgroups(name, context['widget']['value'], attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in optgroups(587)
  for index, (option_value, option_label) in enumerate(self.choices):
/home/user/venv_dd/lib/python3.6/site-packages/django_filters/fields.py in __iter__(252)
  for value in iterable:
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/models.py in __iter__(1137)
  for obj in queryset:

[rpkilby]

Hi @valentijnscholten. Could you possibly open a PR that demonstrates the bug? I've tried testing this with the models provided in the test suite, and have been unable to reproduce the issue. Here's the example test that I added to tests/test_views.py.

class GenericClassBasedViewTests(GenericViewTestCase):
    ...

    def test_queries(self):
        factory = RequestFactory()
        request = factory.get('/')

        class BookFilterSet(FilterSet):
            lovers = ModelMultipleChoiceFilter(queryset=User.objects.all())

            class Meta:
                model = Book
                fields = []

        self.assertIn('lovers', BookFilterSet.base_filters)
        view = FilterView.as_view(filterset_class=BookFilterSet)
        with self.assertNumQueries(2):
            response = view(request)
            response.render()

The above passes as expected, as it makes two queries - one to the books table, one to the users table (Note that I got the queries from changing the number of expected queries above).

Captured queries were:
1. SELECT "tests_user"."id", "tests_user"."username", "tests_user"."first_name", "tests_user"."last_name", "tests_user"."status", "tests_user"."is_active", "tests_user"."is_employed" FROM "tests_user"
2. SELECT "tests_book"."id", "tests_book"."title", "tests_book"."price", "tests_book"."average_rating" FROM "tests_book"

[valentijnscholten]

I could try, it's quite a big project with lots of different things mixed in together. Which might be the cause of the issue if simpler setups like the testcases are working fine.

[rpkilby]

Going to close this for now, as it doesn't look like there's an issue with django-filter itself.

[valentijnscholten]

Did some testing again now that 2.3.0 was out, but the issue remains. However, it's indeed not an issue with django-filter but with django-debug-toolbar. ¯\(ツ)/¯
django-commons/django-debug-toolbar#1239
Queries are only executed once, but reported/displayed twice by django debug toolbar.