Merge pull request #2106 from matrix-org/matthew/1.0/msc688-msc1227-lazy-loading

incorporate LL review from matthew

Author: turt2live

api/client-server/definitions/room_event_filter.yaml

@@ -25,7 +25,8 @@ allOf:
     include_redundant_members:
       type: boolean
       description: |-
-        If ``true``, enables redundant membership events. Does not
+        If ``true``, sends all membership events for all events, even if they have already
+        been sent to the client. Does not
         apply unless ``lazy_load_members`` is ``true``. See
         `Lazy-loading room members <#lazy-loading-room-members>`_
         for more information. Defaults to ``false``.

api/client-server/message_pagination.yaml

@@ -115,15 +115,13 @@ paths:
                 type: array
                 description: |-
                   A list of state events relevant to showing the ``chunk``. For example, if
-                  ``lazy_load_members`` is enabled in the filter then this will contain any
-                  the membership events for the the senders of events in the ``chunk``. Servers should be careful to not exclude
-                  membership events which are older than ones already sent to the client.
-                  Likewise, clients should be cautious and avoid using older membership
-                  events as the current membership event when paginating backwards.
+                  ``lazy_load_members`` is enabled in the filter then this may contain
+                  the membership events for the senders of events in the ``chunk``.
 
-                  Unless ``include_redundant_members`` is ``true``, the server should remove
-                  membership events which would have already been sent to clients in prior calls
-                  to lazy-loading aware endpoints with the same filter.
+                  Unless ``include_redundant_members`` is ``true``, the server
+                  may remove membership events which would have already been
+                  sent to the client in prior calls to this endpoint, assuming
+                  the membership of those members has not changed.
                 items:
                   type: object
                   title: RoomStateEvent

api/client-server/sync.yaml

@@ -39,12 +39,15 @@ paths:
         for more information. Lazy-loading members is only supported on a ``StateFilter``
         for this endpoint. When lazy-loading is enabled, servers MUST include the
         syncing user's own membership event when they join a room, or when the
-        full state of rooms is requested. The user's own membership event is eligible
+        full state of rooms is requested, to aid discovering the user's avatar &
+        displayname.
+
+        Like other members, the user's own membership event is eligible
         for being considered redundant by the server. When a sync is ``limited``,
-        the server MUST return membership events for events in the gap (between ``since`` and the start of the returned timeline),
-        regardless as to whether
-        or not they are redundant. ``include_redundant_members`` is ignored for limited
-        syncs.
+        the server MUST return membership events for events in the gap
+        (between ``since`` and the start of the returned timeline), regardless
+        as to whether or not they are redundant.  This ensures that joins/leaves
+        and profile changes which occur during the gap are not lost.
       operationId: sync
       security:
         - accessToken: []
@@ -149,9 +152,9 @@ paths:
                               type: array
                               description: |-
                                 The users which can be used to generate a room name
-                                if the room does not have one. Required if the room
-                                does not have a ``m.room.name`` or ``m.room.canonical_alias``
-                                state event with non-empty content.
+                                if the room does not have one. Required if the room's
+                                ``m.room.name`` or ``m.room.canonical_alias`` state events
+                                are unset or empty.
 
                                 This should be the first 5 members of the room, ordered
                                 by stream ordering, which are joined or invited. The

specification/client_server_api.rst

@@ -1273,32 +1273,51 @@ Lazy-loading room members
 
 Membership events often take significant resources for clients to track. In an
 effort to reduce the number of resources used, clients can enable "lazy-loading"
-for room members. By doing this, servers will only ever send membership events
+for room members. By doing this, servers will attempt to only send membership events
 which are relevant to the client.
 
-In terms of filters, this means enabling ``lazy_load_members`` on a ``RoomEventFilter``
-(or a ``StateFilter`` in the case of ``/sync`` only). When enabled, lazy-loading
-aware endpoints (see below) will only include membership events for the ``sender``
-of events being included in the response. For example, if a client makes a ``/sync``
-request with lazy-loading enabled, the server will only return membership events
-for the ``sender`` of events in the timeline, not all members of a room.
-
-Repeated calls to lazy-loading aware endpoints will result in redundant membership
-events being excluded by default. Clients often track which membership events they
-already have, therefore making the extra information not as useful to the client.
-Clients can always request redundant membership events by setting ``include_redundant_members``
-to true in the filter.
-
-Servers should be cautious about which events they consider redundant. Membership
-events can change over time, and should be included as relevant to maintain the
-historical record. Likewise, clients should be cautious about treating an older event
-as the current membership event for a user.
-
-.. Note::
-   Repeated calls using the same filter to *any* lazy-loading aware endpoint may
-   result in redundant membership events being excluded from future calls. For example, a
-   request to ``/sync`` followed by a request to ``/messages`` may result in a
-   future call to ``/sync`` excluding membership events returned by the ``/messages`` call.
+It is important to understand that lazy-loading is not intended to be a
+perfect optimisation, and that it may not be practical for the server to
+calculate precisely which membership events are relevant to the client.  As a
+result, it is valid for the server to send redundant membership events to the
+client to ease implementation, although such redundancy should be minimised
+where possible to conserve bandwidth.
+
+In terms of filters, lazy-loading is enabled by enabling ``lazy_load_members``
+on a ``RoomEventFilter`` (or a ``StateFilter`` in the case of ``/sync`` only).
+When enabled, lazy-loading aware endpoints (see below) will only include
+membership events for the ``sender`` of events being included in the response.
+For example, if a client makes a ``/sync`` request with lazy-loading enabled,
+the server will only return membership events for the ``sender`` of events in
+the timeline, not all members of a room.
+
+When processing a sequence of events (e.g. by looping on ``/sync`` or
+paginating ``/messages``), it is common for blocks of events in the sequence
+to share a similar set of senders.  Rather than responses in the sequence
+sending duplicate membership events for these senders to the client, the
+server MAY assume that clients will remember membership events they have
+already been sent, and choose to skip sending membership events for members
+whose membership has not changed.  These are called 'redundant membership
+events'. Clients may request that redundant membership events are always
+included in responses by setting ``include_redundant_members`` to true in the
+filter.
+
+The expected pattern for using lazy-loading is currently:
+
+* Client performs an initial /sync with lazy-loading enabled, and receives
+  only the membership events which relate to the senders of the events it
+  receives.
+* Clients which support display-name tab-completion or other operations which
+  require rapid access to all members in a room should call /members for the
+  currently selected room, with an ``?at`` parameter set to the /sync
+  response's from token. The member list for the room is then maintained by
+  the state in subsequent incremental /sync responses.
+* Clients which do not support tab-completion may instead pull in profiles for
+  arbitrary users (e.g. read receipts, typing notifications) on demand by
+  querying the room state or ``/profile``.
+
+.. TODO-spec
+  This implies that GET /state should also take an ``?at`` param
 
 The current endpoints which support lazy-loading room members are:
 

specification/modules/instant_messaging.rst

@@ -287,7 +287,7 @@ choose a name:
    on the members of the room. Clients should consider `m.room.member`_ events
    for users other than the logged-in user, as defined below.
 
-   i. If the ``m.heroes`` for the room are greater or equal to
+   i. If the number of ``m.heroes`` for the room are greater or equal to
       ``m.joined_member_count + m.invited_member_count - 1``, then use the
       membership events for the heroes to calculate display names for the
       users (`disambiguating them if required`_) and concatenating them. For