Implement per-room rate-limiting for remote joins

[jimmackenzie]

Description:

Synapse includes a few different levers for controlling the flow of actions. We should add a new lever, and limit the number of joins to a room in a given time, including remote joins. I.e. if there has recently been lots of joins in a room (local or remote joins), then servers should ratelimit both local joins and remote attempts to join via /make_join.

By using the existing rate-limit infrastructure, we can tackle some of the issues set out in #12578 around disruptive mass-join events. Rate-limiting helps to prevent performance concerns from handling too many join events at once, and also gives time and space for room moderators to put in place further protections.

Ideally, this limit could be adjusted on a room-by-room basis. Large public rooms would be able to adjust their individual rate-limits, while the default would protect the majority of rooms

[erikjohnston]

Ideally, this limit could be adjusted on a room-by-room basis. Large public rooms would be able to adjust their individual rate-limits, while the default would protect the majority of rooms

This would probably need to be specced. Though I think even having quite a liberal rate limit would help against any join spam issues

[DMRobertson]

What is the proposal here? A per-room rate limit on /make_join, /make_knock, /send_join and /send_knock?

Edit: I ask because Erik talks of spec work---is there a suggestion that this should be enforced across federation?

[DMRobertson]

• track the rate of joins (local, remote) across all servers that we see on a room-by-room basis
• on a request to join/knock, 429 if the threshold for this room has been exceeded
• knocks are similar, but let's just do joins as a first step for now.
  • ideally rate limit knocks too with a separate knob in the future.
• don't ignore incoming remote joins if they would exceed the threshold of join rate
• rate limiter doesn't allow you to update the counters without requesting an action. Change needed there.
• threshold and this behaviour is limited to one server only. on by default with a sensible default value, but a knob to configure it in an emergency. single large global limit for now.

[DMRobertson]

• track the rate of joins (local, remote) across all servers that we see on a room-by-room basis

I think this is hard to do across multiple workers. Maybe we can use redis for this? Idk how much effort it's worth putting into this though(?)

[DMRobertson]

WIP at https://github.com/matrix-org/synapse/tree/dmr/rate-limit-remote-joins

[DMRobertson]

Ideas:

• have each worker snoop on replication traffic so they can see all joins. When we see a join, record it in the rate limiter. Erik cites https://github.com/matrix-org/synapse/blob/bc9b0912cc147713c42e850fbfbb4ee396c8c839/synapse/storage/databases/main/cache.py#L174:L174 for example of snooping
• Don't track anything. Instead, make a DB query asking how many joins there were in the last $DURATION.
  • Cache this so we don't hammer the DB if someone tries a join spam wave? Tricky since it's time-dependent.
  • Have to distinguish between profile changes (join -> join) and brand-new joins (invite -> join).
• Use redis to track joins across multiple workers. Have event persisters write that a join has taken place, and have the workers handling joins read from this.
  • Off the shelf example sounds promising: https://redis.com/redis-best-practices/basic-rate-limiting/ but isn't a 1:1 replacement for our existing limiter. Other examples:
    • https://www.infoworld.com/article/3230455/how-to-use-redis-for-real-time-metering-applications.html
    • https://engineering.classdojo.com/blog/2015/02/06/rolling-rate-limiter/
    • https://blog.callr.tech/rate-limiting-for-distributed-systems-with-redis-and-lua/
  • What about the monolith case?