-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Upgrading from 21.6.3 to latest nightly build removes events/issues from the UI #1401
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
This issue has gone three weeks without activity. In another week, I will close it. But! If you comment or otherwise update it, I will reset the clock, and if you label it "A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀 |
Hi, This is still an issue and I'm afraid to upgrade our production environment with this happening. Please let me know if I need to provide more information. Is there a manual way of migrating data? something like the following?
Thanks, |
@jfabales You can try https://develop.sentry.dev/self-hosted/backup/ Although it has been written on that page:
|
Thanks for the suggestion @aminvakil, given the recommendation on that page, would you recommend doing this approach after upgrading? I'm still keen to find out why I can't see the events after upgrading to a version > 21.6.3, what are the components involved, if they were purged or simply not referenced anymore due to use of different snuba storage (events to errors?) and was there supposed to be a migration step for this? I know it wasn't sentry-cleanup or snuba-cleanup as I removed those services in one of my tests. Is there something I can do (query the api or the database) to verify that the events are still there? |
Sorry for late reply. I'd suggest testing restoring your backup before upgrading, but it's not in a useful state now anyway, so ...
I have never tried messing with database directly... |
Version
21.6.3
Steps to Reproduce
I recently upgraded my sentry self hosted instance from 21.6.3 to the latest nightly release, I think the upgrade went fine as I did not see any errors and all migration results were OK, but now I cannot see the issues on the UI, transactions are still there as far as I can tell and I can post and see new issues on the UI. I feel there's really a simple explanation to this, is this expected?
My upgrade path was 20.7.0 -> 21.6.3 -> latest nightly.
I also tried the following which all ended up the same:
Here are some lines from the snuba and database migration output. I've also attached kafka's logs as I feel they might be relevant.
kafka_logs.txt
▶ Bootstrapping and migrating Snuba ...
Creating sentry_onpremise_zookeeper_1 ...
Creating sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_redis_1 ...
Creating sentry_onpremise_zookeeper_1 ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_redis_1 ... done
Creating sentry_onpremise_kafka_1 ...
Creating sentry_onpremise_kafka_1 ... done
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:07:55,716 Attempting to connect to Kafka (attempt 0)...
2022-03-21 03:07:55,749 Connected to Kafka on attempt 0
2022-03-21 03:07:55,749 Creating Kafka topics...
2022-03-21 03:07:55,873 Topic processed-profiles created
2022-03-21 03:07:55,873 Topic snuba-dead-letter-inserts created
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:08:01,108 Running migration: 0016_drop_legacy_events
/usr/local/lib/python3.8/site-packages/clickhouse_driver/columns/datetimecolumn.py:199: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
local_timezone = get_localzone().zone
2022-03-21 03:08:01,155 Finished: 0016_drop_legacy_events
2022-03-21 03:08:01,163 Running migration: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,204 Finished: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,214 Running migration: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,243 Finished: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,254 Running migration: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,268 Finished: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,278 Running migration: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,296 Finished: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,307 Running migration: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,317 Finished: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,323 Running migration: 0023_polymorphic_repartitioned_bucket_matview
2022-03-21 03:08:01,343 Finished: 0023_polymorphic_repartitioned_bucket_matview
Finished running migrations
▶ Setting up / migrating database ...
Creating sentry_onpremise_postgres_1 ...
Creating sentry_onpremise_memcached_1 ...
Creating sentry_onpremise_symbolicator_1 ...
Creating sentry_onpremise_smtp_1 ...
Creating sentry_onpremise_prometheusexporter_1 ...
Creating sentry_onpremise_snuba-outcomes-consumer_1 ...
Creating sentry_onpremise_snuba-api_1 ...
Creating sentry_onpremise_snuba-replacer_1 ...
Creating sentry_onpremise_snuba-transactions-consumer_1 ...
Creating sentry_onpremise_snuba-consumer_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ...
Creating sentry_onpremise_snuba-sessions-consumer_1 ...
Creating sentry_onpremise_symbolicator_1 ... done
Creating sentry_onpremise_snuba-consumer_1 ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1 ... done
Creating sentry_onpremise_prometheusexporter_1 ... done
Creating sentry_onpremise_smtp_1 ... done
Creating sentry_onpremise_postgres_1 ... done
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... done
Creating sentry_onpremise_snuba-sessions-consumer_1 ... done
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ... done
Creating sentry_onpremise_memcached_1 ... done
Creating sentry_onpremise_snuba-api_1 ... done
Creating sentry_onpremise_snuba-transactions-consumer_1 ... done
Creating sentry_onpremise_snuba-replacer_1 ... done
Creating sentry_onpremise_web_run ...
Creating sentry_onpremise_web_run ... done
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Installing additional dependencies...
...
...
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
03:08:50 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for default
Operations to perform:
Apply all migrations: admin, auth, contenttypes, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying sentry.0220_add_current_release_version_group_resolution... OK
Applying sentry.0221_add_appconnect_upload_dates... OK
Applying sentry.0222_add_datetime_index_to_auditlogentry... OK
Releases: 100% |#################################################| Time: 0:00:00
Applying sentry.0223_semver_backfill_2... OK
Applying sentry.0224_has_sessions_flag... OK
Applying sentry.0225_latest_appconnect_builds_check... OK
Applying sentry.0226_add_visits... OK
Applying sentry.0227_backfill_visits... OK
Applying sentry.0228_update_auditlog_index_with_entry... OK
Applying sentry.0229_drop_jiratenant... OK
Applying sentry.0230_sentry_app_config_jsonfield... OK
Applying sentry.0231_alert_rule_comparison_delta... OK
Releases: 100% |#################################################| Time: 0:00:00
Applying sentry.0232_backfill_missed_semver_releases... OK
Applying sentry.0233_recreate_subscriptions_in_snuba... OK
Applying sentry.0234_grouphistory... OK
Applying sentry.0235_add_metricskeyindexer_table... OK
Applying sentry.0236_remove_legacy_key_transactions... OK
Applying sentry.0237_recreate_subscriptions_in_snuba... OK
Applying sentry.0238_remove_scheduleddeletion_aborted... OK
Applying sentry.0239_drop_scheduleddeletion_aborted... OK
Applying sentry.0240_grouphistory_index... OK
Applying sentry.0241_grouphistory_null_actor... OK
Project Options: 100% |##########################################| Time: 0:00:00
Applying sentry.0242_delete_removed_plugin_data... OK
Repositorys: 100% |##############################################| Time: 0:00:00
Applying sentry.0243_delete_visualstudio_repo_data... OK
Applying sentry.0244_organization_and_integration_foreign_keys... OK
Applying sentry.0245_delete_itunes_credentials... OK
Applying sentry.0246_incident_snapshots_remove_fks... OK
Applying sentry.0247_add_color_column... OK
Applying sentry.0248_add_popularity_column... OK
Applying sentry.0249_add_avatar_type_back... OK
Sentry Apps: 100% |##############################################| Time: 0:00:00
Applying sentry.0250_backfill_popularity... OK
Applying sentry.0251_sentryappavatar_sentryapp_not_unique... OK
Applying sentry.0252_code_mapping_cascade_delete... OK
Applying sentry.0253_add_widget_type... OK
Applying sentry.0254_org_integration_grace_period_end... OK
Applying sentry.0255_delete_code_mappings_with_no_integration... OK
Applying sentry.0256_create_docintegration_table... OK
Applying sentry.0257_add_target_id_and_type_to_integrationfeature... OK
Applying sentry.0258_create_docintegrationavatar_table... OK
Applying sentry.0259_delete_codeowners_and_code_mappings_with_no_integration... OK
Integration Features: 100% |#####################################| Time: 0:00:00
Applying sentry.0260_backfill_integrationfeature... OK
Applying sentry.0261_prepare_remove_sentry_app_column... OK
Applying sentry.0262_drop_sentry_app_from_integrationfeature... OK
Applying sentry.0263_remove_not_null_integrationfeature... OK
Applying sentry.0264_use_booleanfield_docintegration... OK
Applying sentry.0265_add_userrole... OK
Applying sentry.0266_add_dashboard_widget_detail_field... OK
Applying sentry.0267_sentry_release_version_btree... OK
Applying sentry.0268_rename_issue_widget_query_fields... OK
Applying sentry.0269_alertrule_remove_unique_name... OK
Applying sentry.0270_group_history_project_date_added_index... OK
Applying sentry.0271_add_codeowners_auto_sync_setting... OK
Applying sentry.0272_seq_scan_indexes... OK
Applying sentry.0273_fix_grouplink_seqscans... OK
Applying sentry.0274_add_dashboardwidgetquery_columns_aggregates... OK
Applying sentry.0275_rule_fire_history... OK
Applying sentry.0276_rulefirehistory_date_added_index... OK
Applying sentry.0277_backfill_dashboard_widget_query_columns_aggregates... OK
Applying sentry.0278_backfill_codeowners_auto_sync_setting... OK
Applying sentry.0279_add_limit_dashboard_widget... OK
Applying sentry.0280_extend_commit_author_email_length... OK
Applying sentry.0281_add_new_indexer_table... OK
Creating missing DSNs
Correcting Group.num_comments counter
I can provide more details, logs, etc. if needed.
Expected Result
I expect all of the events/issues to show up on the UI (issue view)
Actual Result
Events/Issues don't show up on the UI anymore, even events a day ago don't show up.
New events are successfully posted and show up on the UI but not the existing ones before the upgrade.
The text was updated successfully, but these errors were encountered: