A long queue of to-device messages can prevent outgoing federation working

[shyrwall]

Description

Hi

This will be a vague bug report so I'm hoping someone will come with a "aha" moment when reading this.

For the past week my homeserver has been unable to send outgoing federation messages towards matrix.org. Initially after starting synapse some messages go through but after a few seconds it stops and goes into a retry loop.
During these retries matrix.org , or rather cloudflare, returns a http error of 520.

Upon further inspection i managed to narrow it down to a m.direct_to_device (very large json object) being posted by a single user.
After deleting the event from device_federation_outbox everything worked again.

My theory is that the object was too large so matrix.org/cloudflare threw an error and Synapse just kept retrying.

If this is correct is there a bug in Synapse where it should somehow split this into multiple requests?
Or is it a matrix.org bug that has a low limit on requests size?

Attaching the deleted event.

Thank you
bad_edu.log

Steps to reproduce

Homeserver

xmr.se

Synapse Version

Multiple tested, 1.99 and up. Now 1.103.0

Installation Method

pip (from PyPI)

Database

postgresql. Single, no, no

Workers

Multiple workers

Platform

Not relevant.

Configuration

No response

Relevant log output

2024-03-25 19:22:26,936 - synapse.http.matrixfederationclient - 755 - INFO - federation_transaction_transmission_loop-24 - {PUT-O-29} [matrix.org] Got response headers: 520
2024-03-25 19:22:26,936 - synapse.http.matrixfederationclient - 798 - INFO - federation_transaction_transmission_loop-24 - {PUT-O-29} [matrix.org] Request failed: PUT matrix-federation://matrix.org/_matrix/federation/v1/send/1711389457952: HttpResponseException('520: ')

Anything else that would be useful to know?

No response

[S7evinK]

Did this happen again? Also, in the bad_edu.log, is 704458 the size of the EDU?
#17371 was merged recently, which had a similar symptom to the issue mentioned here. It may be that this is resolved by now.

[shyrwall]

Did this happen again? Also, in the bad_edu.log, is 704458 the size of the EDU? #17371 was merged recently, which had a similar symptom to the issue mentioned here. It may be that this is resolved by now.

Sorry for the late reply. For some reason github emails have been going to my spam folder. It has not happened again. Let's assume it was fixed.

[shyrwall]

Hi. I encountered the problem again in 1.113 and again with matrix.org . Couldn't log unfortunately but fixed by wiping the oldest edu which was ~600kb.

[shyrwall]

Maybe not helpful information but now when it happened again i just checked the size of messages_json is device_federation_outbox and deleted all rows over 300kb. Had about 10 rows between 300-600kb. After deleting all queued messages to matrix.org were completed instantly. I have attached one of the big EDUs.

EDIT: Just realised now that I may be confused with PDU vs EDU :)

Uploading big_edu-240830.log…

[ansiwen]

We also still see this issue (nitro.chat), see #17678 (comment)

The problem seems to be, that limiting the number of events in the EDUs to 50 is not enough, if there are huge events in it. Maybe we should just drop these big events, or are there valid reasons for them?

[shyrwall]

We also still see this issue (nitro.chat), see #17678 (comment)

The problem seems to be, that limiting the number of events in the EDUs to 50 is not enough, if there are huge events in it. Maybe we should just drop these big events, or are there valid reasons for them?

Thanks for commenting. Will try your fix in #17678 next time it happens.