Skip to content

Fix updateCachedShippedBlocks - Thanos Update #4806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

alanprot
Copy link
Member

@alanprot alanprot commented Jul 28, 2022

What this PR does:
Fixing updateCachedShippedBlocks function.

With the thanos update, we started to see the following error:

level=error ts=2022-07-28T02:35:59.814001241Z caller=ingester_v2.go:1753 org_id=blah msg="failed to update cached shipped blocks after shipper initialisation" err="failed to read /data/tsdb/blah/thanos.shipper.json: open /data/tsdb/blah/thanos.shipper.json: no such file or directory"

As thanos changed the error type:

thanos-io/thanos#5174

It does not seems to be a big issue (or not a issue at all) as when the file does not exists there is nothing to update anyway - but the log line is ugly!

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@alanprot alanprot force-pushed the fix/updateCachedShippedBlocks branch from 4ef04f4 to 8afe1f3 Compare July 28, 2022 03:18
@alanprot alanprot force-pushed the fix/updateCachedShippedBlocks branch from 8afe1f3 to ca37194 Compare July 28, 2022 03:19
@alanprot alanprot marked this pull request as ready for review July 28, 2022 03:20
@alanprot alanprot merged commit 427679e into cortexproject:master Jul 28, 2022
@alanprot alanprot deleted the fix/updateCachedShippedBlocks branch July 28, 2022 03:54
alexqyle pushed a commit to alexqyle/cortex that referenced this pull request Aug 2, 2022
alanprot added a commit that referenced this pull request Sep 3, 2022
* Introduced lock file to shuffle sharding grouper

Signed-off-by: Alex Le <[email protected]>

* let redis cache logs log with context (#4785)

* let redis cache logs log with context

Signed-off-by: Mengmeng Yang <[email protected]>

* fix import

Signed-off-by: Mengmeng Yang <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* DoBatch preference to 4xx if error (#4783)

* DoBatch preference to 4xx if error

Signed-off-by: Daniel Blando <[email protected]>

* Fix comment

Signed-off-by: Daniel Blando <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* Updated CHANGELOG and ordered imports

Signed-off-by: Alex Le <[email protected]>

* Fixed lint and removed groupCallLimit

Signed-off-by: Alex Le <[email protected]>

* Changed lock file to json format and make sure planner would not pick up group that is locked by other compactor

Signed-off-by: Alex Le <[email protected]>

* Fix updateCachedShippedBlocks - new thanos (#4806)

Signed-off-by: Alan Protasio <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* Join memberlist on starting with no retry (#4804)

Signed-off-by: Daniel Blando <[email protected]>

* Fix alertmanager log message (#4801)

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* Grafana Cloud uses Mimir now, so remove Grafana Cloud as hosted service in documents (#4809)

* Grafana Cloud uses Mimir, for of Cortex, now

Signed-off-by: Alvin Lin <[email protected]>

* Improve doc

Signed-off-by: Alvin Lin <[email protected]>
Signed-off-by: Alex Le <[email protected]>

* Created block_locker to handle all block lock file operations. Added block lock metrics.

Signed-off-by: Alex Le <[email protected]>

* Moved lock file heart beat into planner and refined planner logic to make sure blocks are locked by current compactor

Signed-off-by: Alex Le <[email protected]>

* Updated documents

Signed-off-by: Alex Le <[email protected]>

* Return concurrency number of group. Use ticker for lock file heart beat

Signed-off-by: Alex Le <[email protected]>

* Renamed lock file to be visit marker file

Signed-off-by: Alex Le <[email protected]>

* Fixed unit test

Signed-off-by: Alex Le <[email protected]>

* Make sure visited block can be picked by compactor visited it

Signed-off-by: Alex Le <[email protected]>

Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Mengmeng Yang <[email protected]>
Signed-off-by: Daniel Blando <[email protected]>
Signed-off-by: Alan Protasio <[email protected]>
Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>
Signed-off-by: Alvin Lin <[email protected]>
Signed-off-by: Alex Le <[email protected]>
Co-authored-by: Mengmeng Yang <[email protected]>
Co-authored-by: Daniel Blando <[email protected]>
Co-authored-by: Alan Protasio <[email protected]>
Co-authored-by: Xiaochao Dong <[email protected]>
Co-authored-by: Alvin Lin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants