Setting a field default removes read-only dot-source fields from serialization when partial=True

[bhomnick]

Checklist

• I have verified that that issue exists against the master branch of Django REST framework.
• I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
• This is not a usage question. (Those should be directed to the discussion group instead.)
• This cannot be dealt with as a third party library. (We prefer new functionality to be in the form of third party libraries where possible.)
• I have reduced the issue to the simplest possible case.
• I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

Steps to reproduce

This seems to be a side-effect of #5375 (documented in #5489) which now requires a default to be set in order for dot-source fields to return None if any part of the relationship is missing.

Here's a simple example:

class Image(models.Model):
    url = models.URLField()
    ...

class Company(models.Model):
    name = models.CharField(max_length=255)
    image = models.ForeignKey('Image', null=True)
    ...

class CompanySerializer(serializers.ModelSerializer):
    image_url = serializer.CharField(read_only=True, default=None, source='image.url')
    
    class Meta:
        model = Company
        fields = ('name, 'image_url')

c  = Company(name='foo')
s = CompanySerializer(c, data={'name': 'bar'}, partial=True)
s.is_valid(raise_exception=True)
s.save()

Expected behavior

I expect s.data will now contain the complete serialized updated instance, including any read-only fields:

>>> s.data 
{'name': 'bar', 'image_url': None}

The relevant line that excludes anything with a default when partial=True: https://github.com/encode/django-rest-framework/blob/master/rest_framework/fields.py#L473

Perhaps this should make an exception for read_only fields?

Actual behavior

Any fields with default set, including read-only, are omitted from the serialized result:

>>> s.data
{'name': 'bar'}

[carltongibson]

Hi @bhomnick,

This looks like expected behaviour. Prior to #5375 your use-case would have raised an AttributeError.

There's a clash between the two flows, serialisation and deserialisation. For serialisation you use data. For deserialisation you pass the data dictionary, use validated_data, and you can also provide the partial flag.

So serialisation:

c  = Company(name='foo')
s = CompanySerializer(c)
data = s.data

And deserialization:

c  = Company(name='foo')
s = CompanySerializer(c, data={'name': 'bar'}, partial=True)
s.is_valid(raise_exception=True)
validated_data = s.validated_data

# s.save() uses `validated_data`

Now, if you're doing a partial update you're probably going to return serializer.data in the response, and this will skip default fields.

In this circumstance, if you need the full representation, then I think you'll need to instantiate a second serialiser with the updated instance and use that for the response. (I'm afraid the combination of dotted source, plus nullable field, plus read-only, plus partial update, plus outgoing representation is just too many things going on at once.)

I don't think there's a change we can make here.

[bhomnick]

@carltongibson Thanks for clarifying. This is definitely changed behavior since the 3.7 upgrade as we now have failing tests because of missing values in PATCH responses that previously appeared.

The only thing I've been able to do to get our test suite passing again is exactly what you suggest--instantiating a second serializer with partial=False. Unfortunately we've now had to modify the built-in mixins.UpdateModelMixin as we rely on it extensively:

def update(self, request, *args, **kwargs):
    partial = kwargs.pop('partial', False)
    instance = self.get_object()
    serializer = self.get_serializer(instance, data=request.data,
                                     partial=partial)
    serializer.is_valid(raise_exception=True)
    self.perform_update(serializer)

    if getattr(instance, '_prefetched_objects_cache', None):
        instance._prefetched_objects_cache = {}

+   serializer = self.get_serializer(instance)
    return response.Response(serializer.data)

I know REST specs don't specifically dictate what a PATCH return value looks like, but to me it seems most intuitive to return the full serialized representation of the updated instance. This doesn't appear possible anymore in our case without writing a custom view. Is this really expected behavior?

[carltongibson]

Yes. It’s expected.

Fields with defaults being skipped for PATCH requests is a long standing behaviour. For dotted sources with nullable parts we need a default. Otherwise we’d have an AttributeError, as per the original issue. So you’re going to hit this.

If you're not keen on adjusting the view code you could always use a field subclass to adjust the get_attribute behaviour to function differently...