Adding OrderingSchema for ordering QuerySets

[aryan-curiel]

Problem

The FilterSchema definition is a simple but effective approach to centralize logic around filtering QuerySets. A similar approach could be followed for ordering.
Current for ordering and filtering in the same handler we would the following:

@api.get("/books")
def list_books(request, filters: BookFilterSchema = Query(...), order_by: list[str] | None = None):
    books = Book.objects.order_by(**order_by)
    books = filters.filter(books)
    return books

But what if we want to limit the API to allow ordering by just some of the fields. Or if we want to customize ordering based on custom fields and logic. What if we want to have a similar approach for other data sources, for example ElasticSearch.

Proposal

This PR propose to include a helper schema class, similar to FilteringSchema, but for ordering. It';s a simple schema class, with only one field: order_by, that accepts a list of string.
The allowed fields can be specified through the Config inner class, and a Pydantic validator will check that the provided query values are part of the allowed fields.
The schema then will provide a .sort() method (similar to .filter()) that we can use to pass the query set, and expect it ordered as a returned value.
The values can be provided using django standard behavior for descending order.

Example

Using it with out-of-the-box definition, allowing all fields from the model.

from ninja import OrderingSchema

@api.get("/books")
def list_books(request, filters: BookFilterSchema = Query(...), ordering: OrderingSchema = Query(...)):
    books = Book.objects.all()
    books = filters.filter(books)
    books = ordering.sort(books)
    return books

Using it with custom definition of allowed fields

from ninja import OrderingSchema

class BookOrderingSchema(OrderingSchema):
    class Meta:
        allowed_fields = ['name', 'created_at']  # Leaving out `author` field

@api.get("/books")
def list_books(request, filters: BookFilterSchema = Query(...), ordering: BookOrderingSchema = Query(...)):
    books = Book.objects.all()
    books = filters.filter(books)
    books = ordering.sort(books)
    return books

Other ideas not followed

I also considered to have a default value field in the config, but decided to go with field default definition on custom schema level
Another consideration was to create a class method factory in the OrderingSchema, so it can be define inline, but I wasn't sure if it would be used:

@api.get("/books")
def list_books(
    request,
    filters: BookFilterSchema = Query(...),
    ordering: OrderingSchema.with_allowed_fields('name', 'created_at') = Query(...)
):
    ...

Notes

I didn't add field validation with Model definition, to keep the practices followed in the FilterSchema definition.
Also, I think is a good idea to keep this helpers class as simple as possible, and give room to personalization for more complex scenarios. However, let me know if validating allowed_fields with Model fields is something we would like to have, and I can update the PR.

This was really useful in a personal project, were we needed to provide different ordering behaviors for a QuerySet and for an OpenSearch query. We had to do similar personalizations for pagination and filtering.

[shijl0925]

How about OrderingSchema.sort support list type sort?

class OrderingSchema(OrderingBaseSchema):
    def sort(self, items: Union[QuerySet, List]) -> Union[QuerySet, List]:
        if self.order_by:
            if isinstance(items, QuerySet):  # type:ignore
                return items.order_by(*self.order_by)
            elif isinstance(items, list) and items:
                def multisort(xs: List, specs: List[Tuple[str, bool]]) -> List:
                    getter = itemgetter if isinstance(xs[0], dict) else attrgetter
                    for key, reverse in specs:
                        xs.sort(key=getter(key), reverse=reverse)
                    return xs

                return multisort(
                    items,
                    [
                        (o[int(o.startswith("-")):], o.startswith("-"))
                        for o in self.order_by
                    ],
                )

        return items

[aryan-curiel]

How about OrderingSchema.sort support list type sort?

Adding support for list type is definitely a good idea. However, not sure adding it to this sort method is the most convenient approach.
For example, if we want to support other collections, then we would need to add another type hint, and another elif.
Any extension for supporting other collections or types, might be more suitable as a subclass of the base OrderingBaseSchema. However, we might need to make some small arrangements in the base class for a better definition.

[shijl0925]

May I ask if there is any progress on this OrderingSchema? I found same failure in these pull request actions. But I can not see these failed logs. What happened about this pull request?

[aryan-curiel]

May I ask if there is any progress on this OrderingSchema?

Honestly, I'm not sure. I haven't received any review, and find no sense in the failed tests related to schemas in the pipeline, so I haven't been able to fix it. If you need it with some level of urgency, feel free to bring the added code to your code base.

[crbunney]

For consideration on this topic, this library also exists and provides ordering support to django-ninja: https://eadwincode.github.io/django-ninja-extra/tutorial/ordering/
I don't know how the functionality & implementation compare to the PR, but thought it might be useful for anyone looking at this to be aware of it

[aryan-curiel]

@crbunney Wow, thanks! Didn't know about it. I'm also doing a library for supporting filtering, ordering and pagination for Ninja and FastAPI with multiple data sources. I'm not a big fan of using decorators for this :( It feels like it;s obfuscating the logic. In any case, it's amazing that we have already something like ninja extras!

[lthon-sha]

Hi @aryan-curiel, just wanted to say a huge thank you for this PR! The OrderingSchema implementation is incredibly clean, elegant. It's exactly the solution I was looking for. Amazing work!

[lthon-sha]

I noticed that when order_by is empty ([]), calling queryset.order_by() clears the model's default ordering from Meta.ordering.

Maybe we should only call order_by() when there are actual fields to sort by?

[aryan-curiel]

I noticed that when order_by is empty ([]), calling queryset.order_by() clears the model's default ordering from Meta.ordering.

Maybe we should only call order_by() when there are actual fields to sort by?

@lthon-sha Thanks for that amazing feedback! I will update the PR according to your recommendation and add the needed tests

[aryan-curiel]

@vitalik I would love a feedback from any of the more experienced contributors. It has been almost a year since the PR was opened, and I would love to move forward with it if it's something that you all consider it might bring some value.

[vitalik]

Hi @aryan-curiel

Sorry for dragging this for this long - but we finally getting rid of "Config" child class in favour of Meta (as it will be "broken" in future pedantic releases

Could you rework Config -> Meta like

django-ninja/ninja/filter_schema.py

Lines 44 to 47 in 52d510a

	class FilterSchema(Schema):
	model_config = ConfigDict(from_attributes=True)
	class Meta:

[aryan-curiel]

Hi @vitalik

That's awesome!
PR updated with the requested changes. Thanks for the feedback.

Let me know if anything else is needed.

[cassandra-aq]

Hi, @vitalik do you have any plan to merge this PR ? It seems like @aryan-curiel has made the change you asked )

[brendan-morin]

This looks great, @aryan-curiel, thank you for putting this PR together!

For the ordering param name, I think because this would be public to end users, we would want to have the query parameter itself be configurable rather than hardcoding to order_by. Use cases could be preferences/requirements to use other field naming/consistency conventions (sort/rank/etc), as well as users who have APIs localized into languages other than English. Having a default order_by seems reasonable, but allowing an override would improve flexibility (perhaps something like the would be possible):

class Meta:
    ordering_query_param = "order_by"
        
...

@model_validator(mode="after")
    def validate_ordering(self) -> "OrderingBaseSchema":
        field_name = getattr(self.Meta, "ordering_query_param")
        value = getattr(self, field_name, [])
        ...
        

Similarly, it may also not always be desirable to expose direct model relationships in ordering to end users, so facilitating some type of mapping of declared filters to publicly facing options would be preferable. Unless I'm misunderstanding, this seems to basically just pass through to the ORM with some optional validation, but if we have a field with a lot of relationships or strange internal naming conventions, we would ideally be able to alias this for end users.

For example, we could have a valid relationship product__variant__brand__formatted_name, but may want to simplify in our api as the query param brand. Perhaps allowing allows_fields to represent a mapping would make sense (or allowed_fields_map if mixed list/dict types feel wrong):

class Meta:
    allowed_fields = {
        "brand": "product__variant__brand__formatted_name"
    }

[aryan-curiel]

@brendan-morin
I updated the PR based on your requests. Please take a look, and let me know your thoughts on the applied changes
Tests and docs also updated accordingly.

[brendan-morin]

This is a clean implementation of a popularly requested feature, looking forward to @vitalik's thoughts if this is good to merge.

[brendan-morin]

nice!

[brendan-morin]

Looks good, this offers some great flexibility. I think the duck-typing approach here to let this be either a dict or list is a good call and probably preferred to the alternate having two more strongly typed configs that could conflict (e.g. allowed_fields, allowed_fields_dict)