Introduced lock file to shuffle sharding grouper #4805

alexqyle · 2022-07-28T01:15:05Z

What this PR does:

Introduced lock file for every blocks are under compaction while using shuffle sharding grouper. The content of lock file is simply a timestamp when the lock file is updated by heart beat process while under compaction. The lock file will make sure blocks are under compaction will not be picked up by other compactor at the same time which fixed Add shuffle-sharding for the compactor #4433 (comment)
Grouper will return the first group with all blocks not having a lock file or timestamp inside lock file is more than 5 minutes old. This replaced the logic using group hash to determine which compactor should own which group. After this change, every compactor can compact groups that belong to the tenant assigned to the compactor and the blocks in the group are not locked.
Incoming groups are still sorted in the same way that smallest range, oldest min time first. The reason only first group would return is that big tenant compaction would not block compaction for other tenant for a long period of time.

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Alex Le <[email protected]>

* let redis cache logs log with context Signed-off-by: Mengmeng Yang <[email protected]> * fix import Signed-off-by: Mengmeng Yang <[email protected]> Signed-off-by: Alex Le <[email protected]>

* DoBatch preference to 4xx if error Signed-off-by: Daniel Blando <[email protected]> * Fix comment Signed-off-by: Daniel Blando <[email protected]> Signed-off-by: Alex Le <[email protected]>

Signed-off-by: Alex Le <[email protected]>

… up group that is locked by other compactor Signed-off-by: Alex Le <[email protected]>

Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Alex Le <[email protected]>

Signed-off-by: Daniel Blando <[email protected]>

@damnever

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> Signed-off-by: Alex Le <[email protected]>

…ce in documents (cortexproject#4809) * Grafana Cloud uses Mimir, for of Cortex, now Signed-off-by: Alvin Lin <[email protected]> * Improve doc Signed-off-by: Alvin Lin <[email protected]> Signed-off-by: Alex Le <[email protected]>

Signed-off-by: Alex Le <[email protected]>

pkg/compactor/shuffle_sharding_grouper.go

pkg/compactor/shuffle_sharding_planner.go

…block lock metrics. Signed-off-by: Alex Le <[email protected]>

pkg/compactor/block_locker.go

pkg/compactor/shuffle_sharding_planner.go

pkg/compactor/shuffle_sharding_grouper.go

…make sure blocks are locked by current compactor Signed-off-by: Alex Le <[email protected]>

Signed-off-by: Alex Le <[email protected]>

pkg/compactor/block_locker.go

Signed-off-by: Alan Protasio <[email protected]>

alanprot · 2022-08-30T22:58:32Z

This is amazing overall! Just added a few nits!

pkg/compactor/shuffle_sharding_grouper.go

Signed-off-by: Alex Le <[email protected]>

pkg/compactor/shuffle_sharding_grouper.go

Signed-off-by: Alex Le <[email protected]>

pull-request-size bot added the size/L label Jul 28, 2022

alexqyle force-pushed the lock-file-for-cortex-compactor branch from 682fd23 to 248eab1 Compare July 28, 2022 01:23

alexqyle marked this pull request as ready for review July 28, 2022 01:28

pull-request-size bot added size/XL and removed size/L labels Aug 2, 2022

alexqyle and others added 10 commits August 2, 2022 10:43

Introduced lock file to shuffle sharding grouper

dc65d95

Signed-off-by: Alex Le <[email protected]>

let redis cache logs log with context (cortexproject#4785)

41e1f08

* let redis cache logs log with context Signed-off-by: Mengmeng Yang <[email protected]> * fix import Signed-off-by: Mengmeng Yang <[email protected]> Signed-off-by: Alex Le <[email protected]>

DoBatch preference to 4xx if error (cortexproject#4783)

20e6502

* DoBatch preference to 4xx if error Signed-off-by: Daniel Blando <[email protected]> * Fix comment Signed-off-by: Daniel Blando <[email protected]> Signed-off-by: Alex Le <[email protected]>

Updated CHANGELOG and ordered imports

735e10d

Signed-off-by: Alex Le <[email protected]>

Fixed lint and removed groupCallLimit

a74e8de

Signed-off-by: Alex Le <[email protected]>

Changed lock file to json format and make sure planner would not pick…

91e381a

… up group that is locked by other compactor Signed-off-by: Alex Le <[email protected]>

Fix updateCachedShippedBlocks - new thanos (cortexproject#4806)

f45290c

Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Alex Le <[email protected]>

Join memberlist on starting with no retry (cortexproject#4804)

9c3efbc

Signed-off-by: Daniel Blando <[email protected]>

Fix alertmanager log message (cortexproject#4801)

275881a

Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> Signed-off-by: Alex Le <[email protected]>

alexqyle force-pushed the lock-file-for-cortex-compactor branch from 9089ea3 to d6d9349 Compare August 2, 2022 17:50

Merge branch 'master' into lock-file-for-cortex-compactor

4714e83

Signed-off-by: Alex Le <[email protected]>

roystchiang reviewed Aug 3, 2022

View reviewed changes

Created block_locker to handle all block lock file operations. Added …

7cf4862

…block lock metrics. Signed-off-by: Alex Le <[email protected]>

roystchiang reviewed Aug 4, 2022

View reviewed changes

pkg/compactor/block_locker.go Outdated Show resolved Hide resolved

pkg/compactor/block_locker.go Outdated Show resolved Hide resolved

pkg/compactor/shuffle_sharding_planner.go Outdated Show resolved Hide resolved

pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved

alexqyle and others added 3 commits August 4, 2022 15:23

Moved lock file heart beat into planner and refined planner logic to …

835251a

…make sure blocks are locked by current compactor Signed-off-by: Alex Le <[email protected]>

Updated documents

3e5f65d

Signed-off-by: Alex Le <[email protected]>

Merge branch 'master' into lock-file-for-cortex-compactor

00ea6da

Signed-off-by: Alex Le <[email protected]>

alanprot reviewed Aug 30, 2022

View reviewed changes

pkg/compactor/block_locker.go Outdated Show resolved Hide resolved

Merge branch 'master' into lock-file-for-cortex-compactor

a5a47b7

Signed-off-by: Alan Protasio <[email protected]>

alanprot reviewed Aug 30, 2022

View reviewed changes

pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved

alexqyle added 3 commits August 30, 2022 17:50

Return concurrency number of group. Use ticker for lock file heart beat

d353004

Signed-off-by: Alex Le <[email protected]>

Renamed lock file to be visit marker file

b7b0663

Signed-off-by: Alex Le <[email protected]>

Fixed unit test

ad5963b

Signed-off-by: Alex Le <[email protected]>

alvinlin123 approved these changes Sep 2, 2022

View reviewed changes

alanprot reviewed Sep 2, 2022

View reviewed changes

pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved

Make sure visited block can be picked by compactor visited it

c47eb5c

Signed-off-by: Alex Le <[email protected]>

alanprot merged commit 546d044 into cortexproject:master Sep 3, 2022

alanprot mentioned this pull request Sep 19, 2022

Ignore not found errors when compacting users #4870

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduced lock file to shuffle sharding grouper #4805

Introduced lock file to shuffle sharding grouper #4805

alexqyle commented Jul 28, 2022 •

edited

Loading

alanprot commented Aug 30, 2022

Introduced lock file to shuffle sharding grouper #4805

Introduced lock file to shuffle sharding grouper #4805

Conversation

alexqyle commented Jul 28, 2022 • edited Loading

alanprot commented Aug 30, 2022

alexqyle commented Jul 28, 2022 •

edited

Loading