Upgrading from 21.6.3 to latest nightly build removes events/issues from the UI

[jfabales]

Version

21.6.3

Steps to Reproduce

I recently upgraded my sentry self hosted instance from 21.6.3 to the latest nightly release, I think the upgrade went fine as I did not see any errors and all migration results were OK, but now I cannot see the issues on the UI, transactions are still there as far as I can tell and I can post and see new issues on the UI. I feel there's really a simple explanation to this, is this expected?
My upgrade path was 20.7.0 -> 21.6.3 -> latest nightly.

I also tried the following which all ended up the same:

increased SENTRY_EVENT_RETENTION_DAYS value
commented out all cleanup services (sentry-cleanup, snuba-cleanup, etc.)
upgraded to 21.12.0

Here are some lines from the snuba and database migration output. I've also attached kafka's logs as I feel they might be relevant.
kafka_logs.txt

▶ Bootstrapping and migrating Snuba ...
Creating sentry_onpremise_zookeeper_1 ...
Creating sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_redis_1 ...
Creating sentry_onpremise_zookeeper_1 ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_redis_1 ... done
Creating sentry_onpremise_kafka_1 ...
Creating sentry_onpremise_kafka_1 ... done
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:07:55,716 Attempting to connect to Kafka (attempt 0)...
2022-03-21 03:07:55,749 Connected to Kafka on attempt 0
2022-03-21 03:07:55,749 Creating Kafka topics...
2022-03-21 03:07:55,873 Topic processed-profiles created
2022-03-21 03:07:55,873 Topic snuba-dead-letter-inserts created
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:08:01,108 Running migration: 0016_drop_legacy_events
/usr/local/lib/python3.8/site-packages/clickhouse_driver/columns/datetimecolumn.py:199: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
local_timezone = get_localzone().zone
2022-03-21 03:08:01,155 Finished: 0016_drop_legacy_events
2022-03-21 03:08:01,163 Running migration: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,204 Finished: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,214 Running migration: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,243 Finished: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,254 Running migration: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,268 Finished: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,278 Running migration: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,296 Finished: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,307 Running migration: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,317 Finished: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,323 Running migration: 0023_polymorphic_repartitioned_bucket_matview
2022-03-21 03:08:01,343 Finished: 0023_polymorphic_repartitioned_bucket_matview
Finished running migrations

▶ Setting up / migrating database ...
Creating sentry_onpremise_postgres_1 ...
Creating sentry_onpremise_memcached_1 ...
Creating sentry_onpremise_symbolicator_1 ...
Creating sentry_onpremise_smtp_1 ...
Creating sentry_onpremise_prometheusexporter_1 ...
Creating sentry_onpremise_snuba-outcomes-consumer_1 ...
Creating sentry_onpremise_snuba-api_1 ...
Creating sentry_onpremise_snuba-replacer_1 ...
Creating sentry_onpremise_snuba-transactions-consumer_1 ...
Creating sentry_onpremise_snuba-consumer_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ...
Creating sentry_onpremise_snuba-sessions-consumer_1 ...
Creating sentry_onpremise_symbolicator_1 ... done
Creating sentry_onpremise_snuba-consumer_1 ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1 ... done
Creating sentry_onpremise_prometheusexporter_1 ... done
Creating sentry_onpremise_smtp_1 ... done
Creating sentry_onpremise_postgres_1 ... done
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... done
Creating sentry_onpremise_snuba-sessions-consumer_1 ... done
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ... done
Creating sentry_onpremise_memcached_1 ... done
Creating sentry_onpremise_snuba-api_1 ... done
Creating sentry_onpremise_snuba-transactions-consumer_1 ... done
Creating sentry_onpremise_snuba-replacer_1 ... done
Creating sentry_onpremise_web_run ...
Creating sentry_onpremise_web_run ... done
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Installing additional dependencies...
...
...
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
03:08:50 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for default
Operations to perform:
Apply all migrations: admin, auth, contenttypes, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying sentry.0220_add_current_release_version_group_resolution... OK
Applying sentry.0221_add_appconnect_upload_dates... OK
Applying sentry.0222_add_datetime_index_to_auditlogentry... OK
Releases: 100% |#################################################| Time: 0:00:00
Applying sentry.0223_semver_backfill_2... OK
Applying sentry.0224_has_sessions_flag... OK
Applying sentry.0225_latest_appconnect_builds_check... OK
Applying sentry.0226_add_visits... OK
Applying sentry.0227_backfill_visits... OK
Applying sentry.0228_update_auditlog_index_with_entry... OK
Applying sentry.0229_drop_jiratenant... OK
Applying sentry.0230_sentry_app_config_jsonfield... OK
Applying sentry.0231_alert_rule_comparison_delta... OK
Releases: 100% |#################################################| Time: 0:00:00
Applying sentry.0232_backfill_missed_semver_releases... OK
Applying sentry.0233_recreate_subscriptions_in_snuba... OK
Applying sentry.0234_grouphistory... OK
Applying sentry.0235_add_metricskeyindexer_table... OK
Applying sentry.0236_remove_legacy_key_transactions... OK
Applying sentry.0237_recreate_subscriptions_in_snuba... OK
Applying sentry.0238_remove_scheduleddeletion_aborted... OK
Applying sentry.0239_drop_scheduleddeletion_aborted... OK
Applying sentry.0240_grouphistory_index... OK
Applying sentry.0241_grouphistory_null_actor... OK
Project Options: 100% |##########################################| Time: 0:00:00
Applying sentry.0242_delete_removed_plugin_data... OK
Repositorys: 100% |##############################################| Time: 0:00:00
Applying sentry.0243_delete_visualstudio_repo_data... OK
Applying sentry.0244_organization_and_integration_foreign_keys... OK
Applying sentry.0245_delete_itunes_credentials... OK
Applying sentry.0246_incident_snapshots_remove_fks... OK
Applying sentry.0247_add_color_column... OK
Applying sentry.0248_add_popularity_column... OK
Applying sentry.0249_add_avatar_type_back... OK
Sentry Apps: 100% |##############################################| Time: 0:00:00
Applying sentry.0250_backfill_popularity... OK
Applying sentry.0251_sentryappavatar_sentryapp_not_unique... OK
Applying sentry.0252_code_mapping_cascade_delete... OK
Applying sentry.0253_add_widget_type... OK
Applying sentry.0254_org_integration_grace_period_end... OK
Applying sentry.0255_delete_code_mappings_with_no_integration... OK
Applying sentry.0256_create_docintegration_table... OK
Applying sentry.0257_add_target_id_and_type_to_integrationfeature... OK
Applying sentry.0258_create_docintegrationavatar_table... OK
Applying sentry.0259_delete_codeowners_and_code_mappings_with_no_integration... OK
Integration Features: 100% |#####################################| Time: 0:00:00
Applying sentry.0260_backfill_integrationfeature... OK
Applying sentry.0261_prepare_remove_sentry_app_column... OK
Applying sentry.0262_drop_sentry_app_from_integrationfeature... OK
Applying sentry.0263_remove_not_null_integrationfeature... OK
Applying sentry.0264_use_booleanfield_docintegration... OK
Applying sentry.0265_add_userrole... OK
Applying sentry.0266_add_dashboard_widget_detail_field... OK
Applying sentry.0267_sentry_release_version_btree... OK
Applying sentry.0268_rename_issue_widget_query_fields... OK
Applying sentry.0269_alertrule_remove_unique_name... OK
Applying sentry.0270_group_history_project_date_added_index... OK
Applying sentry.0271_add_codeowners_auto_sync_setting... OK
Applying sentry.0272_seq_scan_indexes... OK
Applying sentry.0273_fix_grouplink_seqscans... OK
Applying sentry.0274_add_dashboardwidgetquery_columns_aggregates... OK
Applying sentry.0275_rule_fire_history... OK
Applying sentry.0276_rulefirehistory_date_added_index... OK
Applying sentry.0277_backfill_dashboard_widget_query_columns_aggregates... OK
Applying sentry.0278_backfill_codeowners_auto_sync_setting... OK
Applying sentry.0279_add_limit_dashboard_widget... OK
Applying sentry.0280_extend_commit_author_email_length... OK
Applying sentry.0281_add_new_indexer_table... OK
Creating missing DSNs
Correcting Group.num_comments counter

I can provide more details, logs, etc. if needed.

Expected Result

I expect all of the events/issues to show up on the UI (issue view)

Actual Result

Events/Issues don't show up on the UI anymore, even events a day ago don't show up.
New events are successfully posted and show up on the UI but not the existing ones before the upgrade.

[chadwhitacre]

Smells like #1399 and #1252.

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[jfabales]

Hi,

This is still an issue and I'm afraid to upgrade our production environment with this happening. Please let me know if I need to provide more information.

Is there a manual way of migrating data? something like the following?

• export data
• upgrade
• import data

Thanks,
Joseph

[aminvakil]

@jfabales You can try https://develop.sentry.dev/self-hosted/backup/

Although it has been written on that page:

We strongly recommended that you restore your backup on the same version of Sentry on a fresh install (empty database but migrations are run). Otherwise, you are very likely to hit errors and may corrupt your database.

[jfabales]

Thanks for the suggestion @aminvakil, given the recommendation on that page, would you recommend doing this approach after upgrading?

I'm still keen to find out why I can't see the events after upgrading to a version > 21.6.3, what are the components involved, if they were purged or simply not referenced anymore due to use of different snuba storage (events to errors?) and was there supposed to be a migration step for this? I know it wasn't sentry-cleanup or snuba-cleanup as I removed those services in one of my tests.

Is there something I can do (query the api or the database) to verify that the events are still there?

[aminvakil]

Thanks for the suggestion @aminvakil, given the recommendation on that page, would you recommend doing this approach after upgrading?

Sorry for late reply. I'd suggest testing restoring your backup before upgrading, but it's not in a useful state now anyway, so ...

I'm still keen to find out why I can't see the events after upgrading to a version > 21.6.3, what are the components involved, if they were purged or simply not referenced anymore due to use of different snuba storage (events to errors?) and was there supposed to be a migration step for this? I know it wasn't sentry-cleanup or snuba-cleanup as I removed those services in one of my tests.

Is there something I can do (query the api or the database) to verify that the events are still there?

I have never tried messing with database directly...