Skip to content

Add total event, unencrypted message, and e2ee event counts to stats reporting (v2) #18371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MadLittleMods wants to merge 9 commits into from
Closed

Add total event, unencrypted message, and e2ee event counts to stats reporting (v2) #18371

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7f044f9
Add total event, unencrypted message, and e2ee event counts to stats …
anoadragon453 Apr 15, 2025
eb89758
Update changelog PR number
MadLittleMods Apr 29, 2025
a0e94f7
Add `RETURNING` alternative for old versions of SQLite
MadLittleMods Apr 30, 2025
ddf77b5
Always clear the `event_stats` table when the background update starts
MadLittleMods Apr 30, 2025
0b37f9f
Remove superfluous guest from `event_stats` tests
MadLittleMods Apr 30, 2025
29c936f
WIP: Concurrent event insert test
MadLittleMods Apr 30, 2025
3c873f9
Try sharded event_persisters
MadLittleMods Apr 30, 2025
1cb9a0f
Make it more clear that it is a *threadless* thread pool
MadLittleMods May 1, 2025
d0e2a45
Try more things
MadLittleMods May 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/18371.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add `total_event_count`, `total_message_count`, and `total_e2ee_event_count` fields to the homeserver usage statistics.
Copy link
Contributor Author

@MadLittleMods MadLittleMods May 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Workaround

We already report synapse_storage_events_persisted_events and synapse_storage_events_persisted_events_sep metrics which end up with a _total suffix when reported so you will see synapse_storage_events_persisted_events_sep_total for example. Don't be confused by this _total suffix as it's just a Prometheus convention for counters and it tracks cumulative increments, not the absolute event count in Synapse

We additionally have a set of Prometheus rules to collect these metrics as synapse_storage_events_persisted_by_event_type

Which can be presented in Grafana,

Grafana panel JSON 
{
  "datasource": {
    "uid": "$datasource",
    "type": "prometheus"
  },
  "fieldConfig": {
    "defaults": {
      "custom": {
        "lineWidth": 1,
        "fillOpacity": 80,
        "gradientMode": "none",
        "axisPlacement": "auto",
        "axisLabel": "",
        "axisColorMode": "text",
        "axisBorderShow": false,
        "scaleDistribution": {
          "type": "linear"
        },
        "axisCenteredZero": false,
        "hideFrom": {
          "tooltip": false,
          "viz": false,
          "legend": false
        },
        "thresholdsStyle": {
          "mode": "off"
        }
      },
      "color": {
        "mode": "palette-classic"
      },
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "green",
            "value": null
          },
          {
            "color": "red",
            "value": 80
          }
        ]
      },
      "min": 0,
      "unit": "none"
    },
    "overrides": [
      {
        "__systemRef": "hideSeriesFrom",
        "matcher": {
          "id": "byNames",
          "options": {
            "mode": "exclude",
            "names": [
              "m.room.encrypted"
            ],
            "prefix": "All except:",
            "readOnly": true
          }
        },
        "properties": [
          {
            "id": "custom.hideFrom",
            "value": {
              "viz": true,
              "legend": false,
              "tooltip": false
            }
          }
        ]
      }
    ]
  },
  "gridPos": {
    "h": 7,
    "w": 12,
    "x": 12,
    "y": 37
  },
  "id": 46,
  "options": {
    "orientation": "auto",
    "xTickLabelRotation": 0,
    "xTickLabelSpacing": 0,
    "showValue": "auto",
    "stacking": "normal",
    "groupWidth": 0.85,
    "barWidth": 0.97,
    "barRadius": 0,
    "fullHighlight": false,
    "tooltip": {
      "mode": "single",
      "sort": "none"
    },
    "legend": {
      "showLegend": true,
      "displayMode": "list",
      "placement": "bottom",
      "calcs": []
    }
  },
  "pluginVersion": "10.4.1",
  "targets": [
    {
      "datasource": {
        "uid": "$datasource"
      },
      "editorMode": "code",
      "expr": "sum by (type) (increase(synapse_storage_events_persisted_by_event_type{job=~\"$job\",index=~\"$index\",instance=\"$instance\"}[1d]))",
      "format": "time_series",
      "instant": false,
      "intervalFactor": 1,
      "legendFormat": "__auto",
      "refId": "A",
      "step": 20,
      "interval": "1d"
    }
  ],
  "title": "Events per day by Type",
  "type": "barchart"
}

Graph in Grafana showing off daily counts of sent m.room.encrypted messages

If you retain all of your Prometheus data forever, you can then go one step further and create absolute counts from this but it's usually common practice to only persist the last X months of metrics. To get absolute values, you will have to manually keep track and sum these daily counts.

Given that this PR would require further effort to figure out the concurrency problems or a refactor to a different solution, we're going to opt to close it in favor of this workaround. The people having to collate these reports, manually collect data anyway so we will just have them continue with the database queries or these metrics.

Overall, this kind of feature seems fine but turned out to be a lot more effort than originally planned and we don't want to continue if people can live without it for now.

9 changes: 6 additions & 3 deletions docs/usage/administration/monitoring/reporting_homeserver_usage_statistics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,13 @@ The following statistics are sent to the configured reporting endpoint:
| `python_version` | string | The Python version number in use (e.g "3.7.1"). Taken from `sys.version_info`. |
| `total_users` | int | The number of registered users on the homeserver. |
| `total_nonbridged_users` | int | The number of users, excluding those created by an Application Service. |
| `daily_user_type_native` | int | The number of native users created in the last 24 hours. |
| `daily_user_type_native` | int | The number of native, non-guest users created in the last 24 hours. |
| `daily_user_type_guest` | int | The number of guest users created in the last 24 hours. |
| `daily_user_type_bridged` | int | The number of users created by Application Services in the last 24 hours. |
| `total_room_count` | int | The total number of rooms present on the homeserver. |
| `total_event_count` | int | The total number of events present on the homeserver. |
| `total_message_count` | int | The total number of non-state events with type `m.room.message` present on the homeserver. |
| `total_e2ee_event_count` | int | The total number of non-state events with type `m.room.encrypted` present on the homeserver. This can be used as a slight over-estimate for the number of encrypted messages. |
| `daily_active_users` | int | The number of unique users[^1] that have used the homeserver in the last 24 hours. |
| `monthly_active_users` | int | The number of unique users[^1] that have used the homeserver in the last 30 days. |
| `daily_active_rooms` | int | The number of rooms that have had a (state) event with the type `m.room.message` sent in them in the last 24 hours. |
Expand All @@ -50,8 +53,8 @@ The following statistics are sent to the configured reporting endpoint:
| `cache_factor` | int | The configured [`global factor`](../../configuration/config_documentation.md#caching) value for caching. |
| `event_cache_size` | int | The configured [`event_cache_size`](../../configuration/config_documentation.md#caching) value for caching. |
| `database_engine` | string | The database engine that is in use. Either "psycopg2" meaning PostgreSQL is in use, or "sqlite3" for SQLite3. |
| `database_server_version` | string | The version of the database server. Examples being "10.10" for PostgreSQL server version 10.0, and "3.38.5" for SQLite 3.38.5 installed on the system. |
| `log_level` | string | The log level in use. Examples are "INFO", "WARNING", "ERROR", "DEBUG", etc. |
| `database_server_version` | string | The version of the database server. Examples being "10.10" for PostgreSQL server version 10.0, and "3.38.5" for SQLite 3.38.5 installed on the system. |
| `log_level` | string | The log level in use. Examples are "INFO", "WARNING", "ERROR", "DEBUG", etc. |


[^1]: Native matrix users and guests are always counted. If the
Expand Down
41 changes: 36 additions & 5 deletions synapse/app/phone_stats_home.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,22 @@

logger = logging.getLogger("synapse.app.homeserver")

ONE_MINUTE_SECONDS = 60
ONE_HOUR_SECONDS = 60 * ONE_MINUTE_SECONDS

MILLISECONDS_PER_SECOND = 1000

INITIAL_DELAY_BEFORE_FIRST_PHONE_HOME_SECONDS = 5 * ONE_MINUTE_SECONDS
"""
We wait 5 minutes to send the first set of stats as the server can be quite busy the
first few minutes
"""

PHONE_HOME_INTERVAL_SECONDS = 3 * ONE_HOUR_SECONDS
"""
Phone home stats are sent every 3 hours
"""

# Contains the list of processes we will be monitoring
# currently either 0 or 1
_stats_process: List[Tuple[int, "resource.struct_rusage"]] = []
Expand Down Expand Up @@ -121,6 +137,9 @@ async def phone_stats_home(

room_count = await store.get_room_count()
stats["total_room_count"] = room_count
stats["total_event_count"] = await store.count_total_events()
stats["total_message_count"] = await store.count_total_messages()
stats["total_e2ee_event_count"] = await store.count_total_e2ee_events()

stats["daily_active_users"] = common_metrics.daily_active_users
stats["monthly_active_users"] = await store.count_monthly_users()
Expand Down Expand Up @@ -185,12 +204,14 @@ def performance_stats_init() -> None:
# If you increase the loop period, the accuracy of user_daily_visits
# table will decrease
clock.looping_call(
hs.get_datastores().main.generate_user_daily_visits, 5 * 60 * 1000
hs.get_datastores().main.generate_user_daily_visits,
5 * ONE_MINUTE_SECONDS * MILLISECONDS_PER_SECOND,
)

# monthly active user limiting functionality
clock.looping_call(
hs.get_datastores().main.reap_monthly_active_users, 1000 * 60 * 60
hs.get_datastores().main.reap_monthly_active_users,
ONE_HOUR_SECONDS * MILLISECONDS_PER_SECOND,
)
hs.get_datastores().main.reap_monthly_active_users()

Expand All @@ -216,17 +237,27 @@ async def generate_monthly_active_users() -> None:

if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
generate_monthly_active_users()
clock.looping_call(generate_monthly_active_users, 5 * 60 * 1000)
clock.looping_call(
generate_monthly_active_users,
5 * ONE_MINUTE_SECONDS * MILLISECONDS_PER_SECOND,
)
# End of monthly active user settings

if hs.config.metrics.report_stats:
logger.info("Scheduling stats reporting for 3 hour intervals")
clock.looping_call(phone_stats_home, 3 * 60 * 60 * 1000, hs, stats)
clock.looping_call(
phone_stats_home,
PHONE_HOME_INTERVAL_SECONDS * MILLISECONDS_PER_SECOND,
hs,
stats,
)

# We need to defer this init for the cases that we daemonize
# otherwise the process ID we get is that of the non-daemon process
clock.call_later(0, performance_stats_init)

# We wait 5 minutes to send the first set of stats as the server can
# be quite busy the first few minutes
clock.call_later(5 * 60, phone_stats_home, hs, stats)
clock.call_later(
INITIAL_DELAY_BEFORE_FIRST_PHONE_HOME_SECONDS, phone_stats_home, hs, stats
)
4 changes: 4 additions & 0 deletions synapse/rest/client/room.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,8 @@ async def _do(
) -> Tuple[int, JsonDict]:
content = parse_json_object_from_request(request)

logger.info("asdf event send in %s (%s)", room_id, content)

origin_server_ts = None
if requester.app_service:
origin_server_ts = parse_integer(request, "ts")
Expand Down Expand Up @@ -419,6 +421,8 @@ async def _do(
except ShadowBanError:
event_id = generate_fake_event_id()

logger.info("asdf event send DONE in %s (%s) -> %s", room_id, content, event_id)

set_tag("event_id", event_id)
return 200, {"event_id": event_id}

Expand Down
Loading
Loading