Skip to content

Upgrading from 21.6.3 to latest nightly build removes events/issues from the UI #1401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jfabales opened this issue Mar 28, 2022 · 6 comments

Comments

@jfabales
Copy link

Version

21.6.3

Steps to Reproduce

I recently upgraded my sentry self hosted instance from 21.6.3 to the latest nightly release, I think the upgrade went fine as I did not see any errors and all migration results were OK, but now I cannot see the issues on the UI, transactions are still there as far as I can tell and I can post and see new issues on the UI. I feel there's really a simple explanation to this, is this expected?
My upgrade path was 20.7.0 -> 21.6.3 -> latest nightly.

I also tried the following which all ended up the same:

increased SENTRY_EVENT_RETENTION_DAYS value
commented out all cleanup services (sentry-cleanup, snuba-cleanup, etc.)
upgraded to 21.12.0

Here are some lines from the snuba and database migration output. I've also attached kafka's logs as I feel they might be relevant.
kafka_logs.txt

▶ Bootstrapping and migrating Snuba ...
Creating sentry_onpremise_zookeeper_1 ...
Creating sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_redis_1 ...
Creating sentry_onpremise_zookeeper_1 ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_redis_1 ... done
Creating sentry_onpremise_kafka_1 ...
Creating sentry_onpremise_kafka_1 ... done
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:07:55,716 Attempting to connect to Kafka (attempt 0)...
2022-03-21 03:07:55,749 Connected to Kafka on attempt 0
2022-03-21 03:07:55,749 Creating Kafka topics...
2022-03-21 03:07:55,873 Topic processed-profiles created
2022-03-21 03:07:55,873 Topic snuba-dead-letter-inserts created
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
2022-03-21 03:08:01,108 Running migration: 0016_drop_legacy_events
/usr/local/lib/python3.8/site-packages/clickhouse_driver/columns/datetimecolumn.py:199: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
local_timezone = get_localzone().zone
2022-03-21 03:08:01,155 Finished: 0016_drop_legacy_events
2022-03-21 03:08:01,163 Running migration: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,204 Finished: 0014_transactions_remove_flattened_columns
2022-03-21 03:08:01,214 Running migration: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,243 Finished: 0019_aggregate_tables_add_ttl
2022-03-21 03:08:01,254 Running migration: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,268 Finished: 0020_polymorphic_buckets_table
2022-03-21 03:08:01,278 Running migration: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,296 Finished: 0021_polymorphic_bucket_materialized_views
2022-03-21 03:08:01,307 Running migration: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,317 Finished: 0022_repartition_polymorphic_table
2022-03-21 03:08:01,323 Running migration: 0023_polymorphic_repartitioned_bucket_matview
2022-03-21 03:08:01,343 Finished: 0023_polymorphic_repartitioned_bucket_matview
Finished running migrations

▶ Setting up / migrating database ...
Creating sentry_onpremise_postgres_1 ...
Creating sentry_onpremise_memcached_1 ...
Creating sentry_onpremise_symbolicator_1 ...
Creating sentry_onpremise_smtp_1 ...
Creating sentry_onpremise_prometheusexporter_1 ...
Creating sentry_onpremise_snuba-outcomes-consumer_1 ...
Creating sentry_onpremise_snuba-api_1 ...
Creating sentry_onpremise_snuba-replacer_1 ...
Creating sentry_onpremise_snuba-transactions-consumer_1 ...
Creating sentry_onpremise_snuba-consumer_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ...
Creating sentry_onpremise_snuba-sessions-consumer_1 ...
Creating sentry_onpremise_symbolicator_1 ... done
Creating sentry_onpremise_snuba-consumer_1 ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1 ... done
Creating sentry_onpremise_prometheusexporter_1 ... done
Creating sentry_onpremise_smtp_1 ... done
Creating sentry_onpremise_postgres_1 ... done
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... done
Creating sentry_onpremise_snuba-sessions-consumer_1 ... done
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ... done
Creating sentry_onpremise_memcached_1 ... done
Creating sentry_onpremise_snuba-api_1 ... done
Creating sentry_onpremise_snuba-transactions-consumer_1 ... done
Creating sentry_onpremise_snuba-replacer_1 ... done
Creating sentry_onpremise_web_run ...
Creating sentry_onpremise_web_run ... done
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Installing additional dependencies...
...
...
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
03:08:46 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
03:08:50 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for default
Operations to perform:
Apply all migrations: admin, auth, contenttypes, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying sentry.0220_add_current_release_version_group_resolution... OK
Applying sentry.0221_add_appconnect_upload_dates... OK
Applying sentry.0222_add_datetime_index_to_auditlogentry... OK
Releases: 100% |#################################################| Time: 0:00:00
Applying sentry.0223_semver_backfill_2... OK
Applying sentry.0224_has_sessions_flag... OK
Applying sentry.0225_latest_appconnect_builds_check... OK
Applying sentry.0226_add_visits... OK
Applying sentry.0227_backfill_visits... OK
Applying sentry.0228_update_auditlog_index_with_entry... OK
Applying sentry.0229_drop_jiratenant... OK
Applying sentry.0230_sentry_app_config_jsonfield... OK
Applying sentry.0231_alert_rule_comparison_delta... OK
Releases: 100% |#################################################| Time: 0:00:00
Applying sentry.0232_backfill_missed_semver_releases... OK
Applying sentry.0233_recreate_subscriptions_in_snuba... OK
Applying sentry.0234_grouphistory... OK
Applying sentry.0235_add_metricskeyindexer_table... OK
Applying sentry.0236_remove_legacy_key_transactions... OK
Applying sentry.0237_recreate_subscriptions_in_snuba... OK
Applying sentry.0238_remove_scheduleddeletion_aborted... OK
Applying sentry.0239_drop_scheduleddeletion_aborted... OK
Applying sentry.0240_grouphistory_index... OK
Applying sentry.0241_grouphistory_null_actor... OK
Project Options: 100% |##########################################| Time: 0:00:00
Applying sentry.0242_delete_removed_plugin_data... OK
Repositorys: 100% |##############################################| Time: 0:00:00
Applying sentry.0243_delete_visualstudio_repo_data... OK
Applying sentry.0244_organization_and_integration_foreign_keys... OK
Applying sentry.0245_delete_itunes_credentials... OK
Applying sentry.0246_incident_snapshots_remove_fks... OK
Applying sentry.0247_add_color_column... OK
Applying sentry.0248_add_popularity_column... OK
Applying sentry.0249_add_avatar_type_back... OK
Sentry Apps: 100% |##############################################| Time: 0:00:00
Applying sentry.0250_backfill_popularity... OK
Applying sentry.0251_sentryappavatar_sentryapp_not_unique... OK
Applying sentry.0252_code_mapping_cascade_delete... OK
Applying sentry.0253_add_widget_type... OK
Applying sentry.0254_org_integration_grace_period_end... OK
Applying sentry.0255_delete_code_mappings_with_no_integration... OK
Applying sentry.0256_create_docintegration_table... OK
Applying sentry.0257_add_target_id_and_type_to_integrationfeature... OK
Applying sentry.0258_create_docintegrationavatar_table... OK
Applying sentry.0259_delete_codeowners_and_code_mappings_with_no_integration... OK
Integration Features: 100% |#####################################| Time: 0:00:00
Applying sentry.0260_backfill_integrationfeature... OK
Applying sentry.0261_prepare_remove_sentry_app_column... OK
Applying sentry.0262_drop_sentry_app_from_integrationfeature... OK
Applying sentry.0263_remove_not_null_integrationfeature... OK
Applying sentry.0264_use_booleanfield_docintegration... OK
Applying sentry.0265_add_userrole... OK
Applying sentry.0266_add_dashboard_widget_detail_field... OK
Applying sentry.0267_sentry_release_version_btree... OK
Applying sentry.0268_rename_issue_widget_query_fields... OK
Applying sentry.0269_alertrule_remove_unique_name... OK
Applying sentry.0270_group_history_project_date_added_index... OK
Applying sentry.0271_add_codeowners_auto_sync_setting... OK
Applying sentry.0272_seq_scan_indexes... OK
Applying sentry.0273_fix_grouplink_seqscans... OK
Applying sentry.0274_add_dashboardwidgetquery_columns_aggregates... OK
Applying sentry.0275_rule_fire_history... OK
Applying sentry.0276_rulefirehistory_date_added_index... OK
Applying sentry.0277_backfill_dashboard_widget_query_columns_aggregates... OK
Applying sentry.0278_backfill_codeowners_auto_sync_setting... OK
Applying sentry.0279_add_limit_dashboard_widget... OK
Applying sentry.0280_extend_commit_author_email_length... OK
Applying sentry.0281_add_new_indexer_table... OK
Creating missing DSNs
Correcting Group.num_comments counter

I can provide more details, logs, etc. if needed.

Expected Result

I expect all of the events/issues to show up on the UI (issue view)

Actual Result

Events/Issues don't show up on the UI anymore, even events a day ago don't show up.
New events are successfully posted and show up on the UI but not the existing ones before the upgrade.

@chadwhitacre
Copy link
Member

Smells like #1399 and #1252.

@github-actions
Copy link

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@jfabales
Copy link
Author

Hi,

This is still an issue and I'm afraid to upgrade our production environment with this happening. Please let me know if I need to provide more information.

Is there a manual way of migrating data? something like the following?

  • export data
  • upgrade
  • import data

Thanks,
Joseph

@aminvakil
Copy link
Collaborator

@jfabales You can try https://develop.sentry.dev/self-hosted/backup/

Although it has been written on that page:

We strongly recommended that you restore your backup on the same version of Sentry on a fresh install (empty database but migrations are run). Otherwise, you are very likely to hit errors and may corrupt your database.

@jfabales
Copy link
Author

Thanks for the suggestion @aminvakil, given the recommendation on that page, would you recommend doing this approach after upgrading?

I'm still keen to find out why I can't see the events after upgrading to a version > 21.6.3, what are the components involved, if they were purged or simply not referenced anymore due to use of different snuba storage (events to errors?) and was there supposed to be a migration step for this? I know it wasn't sentry-cleanup or snuba-cleanup as I removed those services in one of my tests.

Is there something I can do (query the api or the database) to verify that the events are still there?

@aminvakil
Copy link
Collaborator

Thanks for the suggestion @aminvakil, given the recommendation on that page, would you recommend doing this approach after upgrading?

Sorry for late reply. I'd suggest testing restoring your backup before upgrading, but it's not in a useful state now anyway, so ...

I'm still keen to find out why I can't see the events after upgrading to a version > 21.6.3, what are the components involved, if they were purged or simply not referenced anymore due to use of different snuba storage (events to errors?) and was there supposed to be a migration step for this? I know it wasn't sentry-cleanup or snuba-cleanup as I removed those services in one of my tests.

Is there something I can do (query the api or the database) to verify that the events are still there?

I have never tried messing with database directly...

@getsentry getsentry locked and limited conversation to collaborators May 13, 2022
@chadwhitacre chadwhitacre converted this issue into a discussion May 13, 2022
@getsentry getsentry unlocked this conversation Jun 17, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Jul 3, 2022
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants