General Thread (Q1 2022)

[chadwhitacre]

This is a thread for meta discussion about self-hosted Sentry (SHS).

Please file a ticket if you have a support request (or feature suggestion).

← Q4 2021

[djmaze]

Okay, I'll post this here. The on-premise documentation points to the forum for discussions, but it's not possible to create new topics there. Also, there is supposed to be a pinned post in the forum which explains that people should use Github issues now. But its not visible to me there:

It took me quite some time to figure this out. Could you pin the post in the forums or point the documentations to Github issues directly?

[chadwhitacre]

@djmaze Thanks for letting us know and sorry for the frustration! Glad you found us. :)

I made getsentry/team-ospo#27 to track link cleanup, I had that in a private TODO, sorry I didn't get to it in time. :(

I'm not sure what's up with pinning on Discourse. I see on the post that it says "PINNED ON NOV 18, '21":

But yeah I don't see it pinned anymore. 🤔

Maybe we're running afoul of this "feature"?

[chadwhitacre]

I unchecked both of these but still don't see the item pinned as expected.

[chadwhitacre]

lolsob in incognito I get another pin back but not the one we want

[chadwhitacre]

I just bumped it. ¯\_(ツ)_/¯

[djmaze]

Did you try unpinning and pinning it again? ;)

[chadwhitacre]

Aka turn it off and turn it back on again? :-P

Done, here's hoping it works ...

[djmaze]

Seems to work. On the topic page, it says:

This topic is pinned for you; it will display at the top of its category

Thanks!

[bergpb]

Hello for all, here is the status of the following attempt: #1175 (comment)

After increasing the value of the variable SENTRY_EVENT_RETENTION_DAYS and upgrading again to version 21.5 the same behavior occurred, most of the issues were lost. For example, in project A there were more than 4000 issues in version 9.0, but after upgrading to version 21.5 only 535 are registered. Also, the service to auto-remove the old issues was removed from docker-compose file. Does anyone have another tip to try?

[aminvakil]

Hello for all, here is the status of the following attempt: #1175 (comment)

After increasing the value of the variable SENTRY_EVENT_RETENTION_DAYS and upgrading again to version 21.5 the same behavior occurred, most of the issues were lost. For example, in project A there were more than 4000 issues in version 9.0, but after upgrading to version 21.5 only 535 are registered. Also, the service to auto-remove the old issues was removed from docker-compose file. Does anyone have another tip to try?

Not sure what has happened in your situation, but did you first upgrade to 9.1.2 as it has been told on https://develop.sentry.dev/self-hosted/releases/#hard-stops prior to upgrading to 21.5.0 ?

Because missing that step may arise some issues.

[bergpb]

Yeah, on my first try I'm follow the hard stops, but I'm found some problems in migrations described here:
https://forum.sentry.io/t/migration-errors-9-1-2-to-21-6-1-project-object-has-no-attribute-get-option/14279/4
After that I worked on the upgrade using these steps: 9.0 > 9.1 > 20.12.1 > 21.5, and with this, I don't have errors with migrations, but lost some issues in step to upgrade to 9.1 > 20.12.1.

[chadwhitacre]

lost some issues in step to upgrade to 9.1 > 20.12.1

Do you lose issues if you go from 9.1.2 straight to 21.5.0? There may be bugs in migrations in earlier versions.

[bergpb]

Yes @chadwhitacre when I try to go from version 9.1 directly to version 21.5 I found some errors in migration and lost issues too.

An error occured while trying to migrate the following event: <sentry.eventstore.models.Event object at 0x7ff941841ba8>
Project' object has no attribute 'get_option'

[chadwhitacre]

Hmmm ... @bergpb this is getting involved enough that we should open a separate ticket to help you work through your scenario. Can you open a new ticket, please?

[bergpb]

Sure @chadwhitacre

[djmaze]

Another question. We realized that the sentry js client often sends client_report requests upon page load, which the server responds to with a 400 response. Looking into the self-hosted documentation and issues, I can't find any mention of it. Is that feature only available in the commercial product?

Would be good to have this cleared up in the documentation as well.

[chadwhitacre]

In general self-hosted has all the same features as SaaS.

Looking into the self-hosted documentation and issues, I can't find any mention of it.

Sorry, I'm lost. What would make sense to mention about client_report requests in the self-hosted documentation that is not already mentioned in the client reports documentation?

[djmaze]

Ah. I dug a bit deeper and found that it's supported since version 21.9.0 in the relay. Our instance is just too old.

Sorry for the noise.

(Minor point that it is difficult to see all changes in a release since the detailed changelogs are spread over all of the service repos.)

[chadwhitacre]

Minor point

Fair point. :) I updated the GH release notes to include links to Sentry, Snuba, and Relay. Do you think we should link to Symbolicator? Anything else I'm forgetting?

I also added a mention of Postgres 10 while I was there.

[djmaze]

@chadwhitacre Excellent, very helpful!

[djmaze]

Not even sure what symbolicator is for and if it is important. If it is, then it might make sense to add it as well.

[chadwhitacre]

Symbolicator is a service we maintain, but it is lower-level and not coupled tightly to Sentry, hence it uses SemVer rather than CalVer and is released on its own schedule, not monthly as with self-hosted and the other components. Let's leave it off. :)

[agorkiy]

Hi everyone. After installing sentry on-prem and launching it with compose, I see that 30 docker pods were launched: - including kafka, zookeper, snuba, etc. Does sentry really needs all those services? I need kind of "minimal" setup, so I was hoping that all sentry setup will be 1 nginx, 1 container with sentry and 1 db. So, question is, can default setup be modified and stripped to some minimal working set of only required containers?

[chadwhitacre]

Short answer is yes, Sentry really needs all those services.

Sounds like a 👍 for #789. 😌

[agorkiy]

@chadwhitacre Ok, got it, thanks. Another question - what should be done for setup google SSO? Already added auth-google.client-id and auth-google.client-secret to sentry/config.yml and restarted all stack - but it it didn't take any effect. Anything I am missing here?

[chadwhitacre]

Seems like you found the docs. If you're having trouble go ahead and open a new ticket with detailed info about your environment and the steps you're following and maybe something will come to light there.

[anthonyvelazquez]

Hello, I'm encountering a problem setting up the most recent version of Sentry with an external Postgres DB. We are currently using Vagrant and Puppet to provision everything and with Sentry's default settings, it works perfectly. To use the external DB, I replaced sentry.conf.example.py with a copy of the same file and the DATABASES field updated with the appropriate settings and then it runs the ./install.sh script as normal.
After Puppet finishes provisioning the system, I SSH into the box and see that the conf settings are there but I don't see any changes that occurred to the external DB and I never receive any error messages.

Are there any additional steps required to used an external DB rather than the internal container?

[ppolewicz]

Hello, I've been a self-hosted sentry user for a couple of years but upgraded to the newest version a few weeks ago. I have an 8GB RAM machine with only sentry and a monitoring agent running on it. It keeps running out of memory.

I ran top and have noticed a relay service which I think I don't need. I do not need a special layer for PII scrubbing (if anyone would be scrubbing it for us, it would be the sdks) and we won't ever send anything to sentry.io.

Can I safely shut down the relay service and remove it from docker-compose to save memory and CPU?

[ppolewicz]

Separately, memory usage seems to increase over time. Is this normal?

service	10d ago	today	after restart
relay	707	965	662
zookeeper	097	378	074

Values in megabytes in the table, in gigabytes on the chart

[chadwhitacre]

Can I safely shut down the relay service and remove it from docker-compose to save memory and CPU?

I mean, the official answer is no, relay is a core part of the system ... but go for it, if you get it to work let us know in case others want to try. :)

8GB

This is the bare minimum, I recall a thread somewhere where we had some better numbers tied to event volume, like so many GBs per N events/mo or whatever. If we dug that up it'd be helpful to update the README with it.

On the gradual growth question I'm not sure what the expected upper limit is, maybe @taxilian @kanadaj have some insight?

[chadwhitacre]

see that the conf settings are there

@anthonyvelazquez You're depending on install.sh to copy sentry.conf.example.py to sentry.conf.py, yes? The config values shows up in the latter?

[BorisMan9]

Hi, I didn't think that this deserves it's own ticket so asking here:
We set up self hosted sentry and we "outsourced" Kafka to AWS MSK (3 brokers). Very quickly we started seeing very high disk usage by one of our brokers (nearly 90%, 1000gb of disk) while the other two were using very small disk space (around 6%).

So I started digging into the sentry Kafka topics to try balance the load between the brokers using the kafka-reassign-partitions.sh. tool. I understood that I cant tell which topic could be the "heaviest" - and should be reassigned to a different broker. I also couldn't find info on it in the docs or any other place, so I will be more than happy if someone can elaborate on the topics that potentially can consume a lot of disk space. Here is the list of relevant topics:

• events-subscription-results
• ingest-attachments
• ingest-events
• ingest-transactions
• ingest-sessions
• transaction-subscription-results
• outcomes
• cdc
• event-replacements
• event-replacements-legacy
• events
• metrics-subscriptions-results
• scheduled-subscriptions-events
• scheduled-subscriptions-metrics
• scheduled-subscriptions-sessions
• scheduled-subscriptions-transactions
• sessions-subscriptions-results
• snuba-commit-log
• snuba-metrics-commit-log
• snuba-metrics
• snuba-queries
• snuba-session-commit-log

Thx

P.S
I'm aware of all the log retention configs and how it can get the disk space bloated. Yet I want to understand the correct balance of topics between brokers

[prismic]

I am looking for a workaround for a permissions issue we are running into.

Our use case for Sentry is to spin up an instance where other developers in our organization can go in and manage their own teams and projects, with as little administrative assistance as possible from the Owner role. Based on our testing, running in a single org mode, it seems Admins offers the closest role for what we want: able to add/approve other members into the team, create project in their team, and no permissions to modify any team that they aren't a member of. Great.

A problem arises in the integrations. It seems while Admins are mostly restricted to their own teams, they are somehow able to delete integrations for the entire organization. This seems to be at odds with the rest of the Admin's permissions, since they aren't able to modify other team's configurations, yet are able to delete them.

We thought about two workarounds. The first is to make everyone Member and make it open membership, but we don't want any users to freely join other teams without permission and look at logs that could potentially be sensitive, so it was a no go for us.

The second is to go with multi-org, and create an org for every team, which is too high-maintenance, because any configuration that are the same must now be replicated every org, such as authentication, slack, Github, etc.

Is there a way to change the behavior so that Admins cannot add/delete org-level integrations, but able to manage their own? If we can make it so they must work with an Owner for integration settings, that's okay too, but I am hoping for something in between.

[chadwhitacre]

@BorisMan9 @prismic Can you create new tickets with your questions? This thread is getting too cluttered, I've updated the description to reserve this thread for meta discussion vs. support requests.

[schmitch]

btw. this has been asked over and over again, but I'm still asking, will there be a time where sentry would use something different than docker-volumes and docker-compose. I would really prefer a more modern solution like helm/kustomize or even a solution based on hashicorp. I'm not even asking for a multi host solution, I'm just not the biggest fan of docker-volumes, especially not the way how backing up this stuff. we self-hosted users got really really screwed over after 9.x with these changes since at the moment our backup tooling needs to backup the whole vm (before we only needed to backup postgresql) and docker-compose wil basically never get a really good backup solution since docker-compose was never meant to be used like sentry is using it (granted it works).

I know we are mostly "free" users of your software. but heck I would gladly pay if I can still use my self hosted stuff and have a "better" solution than "we have a shell script here, that should work" (something that can be better integrated with ci/cd and/or other tooling...) and a proper backup solution (I don't care if it still uses docker-compose than underneath, if it works and I can use s3/whatever to back it up easily, does not need to be too much configurable, but it should work)

[taxilian]

@schmitch it's worth noting that you can do that, you just need to set some of it up yourself.

There is a helm chart for setting up sentry which I started with and now I've migrated off of the helm chart and just manage things myself, but it's definitely doable.

[djmaze]

Although I know it is not very popular, but it is quite easy to migrate the compose config to a swarm config and run Sentry on a Docker swarm cluster. We are doing this. You just need a solution for data persistence (using a volume driver).

[kanadaj]

Oh damn I just saw that I got tagged here.

Can I safely shut down the relay service and remove it from docker-compose to save memory and CPU?

I don't believe so, not with the current version of Sentry. Relay in integrated mode AFAIK does some pre-processing that's strictly required for the current Sentry pipeline, which makes the name a bit of a misnomer.

On the gradual growth question I'm not sure what the expected upper limit is, maybe @taxilian @kanadaj have some insight?

My Relay is using 1.62GB and seems mostly settled for the past 2 days. Zookeeper is at 395.3MB but that one depends a lot on the type and degree of workload you have since Kafka stores the metadata in ZK. Total memory usage is at 13GB with some ~3 hour cycles, most of which is Kafka with 5GB memory used. And the faster you ingest, the more memory Kafka needs. I'd dare say 8GB is simply not enough for Sentry at this point, but some periodic cleaning of Kafka and ZK might keep usage down at 8GB, just barely.

[chadwhitacre]

@kanadaj Hey, you are on GitHub! 😅

[ppolewicz]

I'm restarting leaky services (zookeper and relay) every night and this keeps my memory usage below the alert threshold (85% of 8GB).

[jfabales]

Hi,

I hope this is the right place for this question. I recently upgraded my sentry self hosted instance from 21.6.3 to the latest nightly release, I think the upgrade went fine as I did not see any errors and all migration results were OK, but now I cannot see the issues on the UI, transactions are still there as far as I can tell and I can post and see new issues on the UI. I feel there's really a simple explanation to this, is this expected?
My upgrade path was 20.7.0 -> 21.6.3 -> latest by the way and I'm using the latest copy of the docker-compose.yml file.

I also tried the following which all ended up the same:

• increased SENTRY_EVENT_RETENTION_DAYS value
• commented out all cleanup services (sentry-cleanup, snuba-cleanup, etc.)
• upgraded to 21.12.0

Here are some lines from the snuba and database migration output. I've also attached kafka's logs as I feel they might be relevant.
kafka_logs.txt

▶ Bootstrapping and migrating Snuba ...
Creating sentry_onpremise_zookeeper_1 ... 
Creating sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_redis_1      ... 
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_kafka_1      ... 
Creating sentry_onpremise_kafka_1      ... done
Creating sentry_onpremise_snuba-api_run ... 
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:07:55,716 Attempting to connect to Kafka (attempt 0)...
2022-03-21 03:07:55,749 Connected to Kafka on attempt 0
2022-03-21 03:07:55,749 Creating Kafka topics...
2022-03-21 03:07:55,873 Topic processed-profiles created
2022-03-21 03:07:55,873 Topic snuba-dead-letter-inserts created
Creating sentry_onpremise_snuba-api_run ... 
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:08:01,108 Running migration: 0016_drop_legacy_events
/usr/local/lib/python3.8/site-packages/clickhouse_driver/columns/datetimecolumn.py:199: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
  local_timezone = get_localzone().zone
2022-03-21 03:08:01,155 Finished: 0016_drop_legacy_events
2022-03-21 03:08:01,163 Running migration: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,204 Finished: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,214 Running migration: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,243 Finished: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,254 Running migration: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,268 Finished: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,278 Running migration: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,296 Finished: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,307 Running migration: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,317 Finished: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,323 Running migration: 0023_polymorphic_repartitioned_bucket_matview
2022-03-21 03:08:01,343 Finished: 0023_polymorphic_repartitioned_bucket_matview
Finished running migrations

▶ Setting up / migrating database ...
Creating sentry_onpremise_postgres_1 ... 
Creating sentry_onpremise_memcached_1 ... 
Creating sentry_onpremise_symbolicator_1 ... 
Creating sentry_onpremise_smtp_1         ... 
Creating sentry_onpremise_prometheusexporter_1 ... 
Creating sentry_onpremise_snuba-outcomes-consumer_1 ... 
Creating sentry_onpremise_snuba-api_1               ... 
Creating sentry_onpremise_snuba-replacer_1          ... 
Creating sentry_onpremise_snuba-transactions-consumer_1 ... 
Creating sentry_onpremise_snuba-consumer_1              ... 
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ... 
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... 
Creating sentry_onpremise_snuba-sessions-consumer_1                  ... 
Creating sentry_onpremise_symbolicator_1                             ... done
Creating sentry_onpremise_snuba-consumer_1                           ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1                  ... done
Creating sentry_onpremise_prometheusexporter_1                       ... done
Creating sentry_onpremise_smtp_1                                     ... done
Creating sentry_onpremise_postgres_1                                 ... done
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... done
Creating sentry_onpremise_snuba-sessions-consumer_1                  ... done
Creating sentry_onpremise_snuba-subscription-consumer-events_1       ... done
Creating sentry_onpremise_memcached_1                                ... done
Creating sentry_onpremise_snuba-api_1                                ... done
Creating sentry_onpremise_snuba-transactions-consumer_1              ... done
Creating sentry_onpremise_snuba-replacer_1                           ... done
Creating sentry_onpremise_web_run                                    ... 
Creating sentry_onpremise_web_run                                    ... done
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Installing additional dependencies...
...
...
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
03:08:50 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for default
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying sentry.0220_add_current_release_version_group_resolution... OK
  Applying sentry.0221_add_appconnect_upload_dates... OK
  Applying sentry.0222_add_datetime_index_to_auditlogentry... OK
Releases: 100% |#################################################| Time: 0:00:00
  Applying sentry.0223_semver_backfill_2... OK
  Applying sentry.0224_has_sessions_flag... OK
  Applying sentry.0225_latest_appconnect_builds_check... OK
  Applying sentry.0226_add_visits... OK
  Applying sentry.0227_backfill_visits... OK
  Applying sentry.0228_update_auditlog_index_with_entry... OK
  Applying sentry.0229_drop_jiratenant... OK
  Applying sentry.0230_sentry_app_config_jsonfield... OK
  Applying sentry.0231_alert_rule_comparison_delta... OK
Releases: 100% |#################################################| Time: 0:00:00
  Applying sentry.0232_backfill_missed_semver_releases... OK
  Applying sentry.0233_recreate_subscriptions_in_snuba... OK
  Applying sentry.0234_grouphistory... OK
  Applying sentry.0235_add_metricskeyindexer_table... OK
  Applying sentry.0236_remove_legacy_key_transactions... OK
  Applying sentry.0237_recreate_subscriptions_in_snuba... OK
  Applying sentry.0238_remove_scheduleddeletion_aborted... OK
  Applying sentry.0239_drop_scheduleddeletion_aborted... OK
  Applying sentry.0240_grouphistory_index... OK
  Applying sentry.0241_grouphistory_null_actor... OK
Project Options: 100% |##########################################| Time: 0:00:00
  Applying sentry.0242_delete_removed_plugin_data... OK
Repositorys: 100% |##############################################| Time: 0:00:00
  Applying sentry.0243_delete_visualstudio_repo_data... OK
  Applying sentry.0244_organization_and_integration_foreign_keys... OK
  Applying sentry.0245_delete_itunes_credentials... OK
  Applying sentry.0246_incident_snapshots_remove_fks... OK
  Applying sentry.0247_add_color_column... OK
  Applying sentry.0248_add_popularity_column... OK
  Applying sentry.0249_add_avatar_type_back... OK
Sentry Apps: 100% |##############################################| Time: 0:00:00
  Applying sentry.0250_backfill_popularity... OK
  Applying sentry.0251_sentryappavatar_sentryapp_not_unique... OK
  Applying sentry.0252_code_mapping_cascade_delete... OK
  Applying sentry.0253_add_widget_type... OK
  Applying sentry.0254_org_integration_grace_period_end... OK
  Applying sentry.0255_delete_code_mappings_with_no_integration... OK
  Applying sentry.0256_create_docintegration_table... OK
  Applying sentry.0257_add_target_id_and_type_to_integrationfeature... OK
  Applying sentry.0258_create_docintegrationavatar_table... OK
  Applying sentry.0259_delete_codeowners_and_code_mappings_with_no_integration... OK
Integration Features: 100% |#####################################| Time: 0:00:00
  Applying sentry.0260_backfill_integrationfeature... OK
  Applying sentry.0261_prepare_remove_sentry_app_column... OK
  Applying sentry.0262_drop_sentry_app_from_integrationfeature... OK
  Applying sentry.0263_remove_not_null_integrationfeature... OK
  Applying sentry.0264_use_booleanfield_docintegration... OK
  Applying sentry.0265_add_userrole... OK
  Applying sentry.0266_add_dashboard_widget_detail_field... OK
  Applying sentry.0267_sentry_release_version_btree... OK
  Applying sentry.0268_rename_issue_widget_query_fields... OK
  Applying sentry.0269_alertrule_remove_unique_name... OK
  Applying sentry.0270_group_history_project_date_added_index... OK
  Applying sentry.0271_add_codeowners_auto_sync_setting... OK
  Applying sentry.0272_seq_scan_indexes... OK
  Applying sentry.0273_fix_grouplink_seqscans... OK
  Applying sentry.0274_add_dashboardwidgetquery_columns_aggregates... OK
  Applying sentry.0275_rule_fire_history... OK
  Applying sentry.0276_rulefirehistory_date_added_index... OK
  Applying sentry.0277_backfill_dashboard_widget_query_columns_aggregates... OK
  Applying sentry.0278_backfill_codeowners_auto_sync_setting... OK
  Applying sentry.0279_add_limit_dashboard_widget... OK
  Applying sentry.0280_extend_commit_author_email_length... OK
  Applying sentry.0281_add_new_indexer_table... OK
Creating missing DSNs
Correcting Group.num_comments counter

I can provide more details, logs, etc. if needed.

Thank you in advance.

Cheers!
Joseph

[chadwhitacre]

@jfabales Please file a new ticket with your support request. Thanks.

(P.S. Sorry for the late reply, I was OOO.)

[jfabales]

Thank you @chadwhitacre and no worries.
Here's the link to the new issue I posted.

Thanks again,
Joseph

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[chadwhitacre]

Gonna let the clock run out on this one and then not open another, since the signal to noise has been really low with these.