Help to recover from corrupted levelqueue #24912
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GiteaBot
added
the
lgtm/need 2
wxiaoguang
force-pushed
the
help-corrupted-levelquque
branch
3 times, most recently
from
9b054c7 to
eb0570f
Compare
wxiaoguang
force-pushed
the
help-corrupted-levelquque
branch
from
eb0570f to
9cd00ad
Compare
lunny
approved these changes
May 25, 2023
GiteaBot
added
lgtm/need 1
KN4CK3R
reviewed
May 25, 2023
| internal *levelqueue.UniqueQueue | ||
| conn string | ||
| cfg *BaseConfig | ||
| internal atomic.Pointer[levelqueue.UniqueQueue] |
There was a problem hiding this comment.
Because RemoveAll needs to re-create the levelqueue. The levelqueue is used by different goroutines.
| assert.Len(t, keys, 5) | ||
|
|
||
| // delete the queue item key, to corrupt the queue | ||
| assert.NoError(t, db.Delete(itemKey, nil)) |
There was a problem hiding this comment.
Why does deleting an item corrupt the queue?
There was a problem hiding this comment.
See this PR's description:
if the keys in LevelDB went out-of-sync, the LevelQueue itself doesn't have the ability to recover, eg:
- LevelQueue.Len() reports 100
- LevelQueue.LPop() reports ErrNotFound = errors.New("no key found")
It's levelqueue's bug, not Gitea's. The levelqueue.LPop was written as this.
Yes, delete all corrupted data in leveldb, and re-create the levelqueue. |
silverwind
approved these changes
May 28, 2023
GiteaBot
added
lgtm/done
lunny
added
the
reviewed/wait-merge
GiteaBot
removed
the
reviewed/wait-merge
zjjhot
added a commit
to zjjhot/gitea
that referenced
this pull request
May 29, 2023
* upstream/main: Test query must have "order by" explicitly to avoid unstable results (go-gitea#24963) Help to recover from corrupted levelqueue (go-gitea#24912) [skip ci] Updated translations via Crowdin Remove meta tags `theme-color` and `default-theme` (go-gitea#24960) Add dark mode to API Docs (go-gitea#24971) Update JS dependencies (go-gitea#24969) Replace Fomantic reset module with our own (go-gitea#24948) simple docs fixes: 'pull request' page (en-us & zh-tw) link path to 'issue-pull-request-templates' (go-gitea#24961) Remove reference to caddy v1 in docs (go-gitea#24962) Improve and fix bugs surrounding reactions (go-gitea#24760) Use `[git.config]` for reflog cleaning up (go-gitea#24958) Improve logger Pause handling (go-gitea#24946) Do not output "Trace" level logs from process manager by default (go-gitea#24952) Make the 500 page load themes (go-gitea#24953) [skip ci] Updated translations via Crowdin docs: remove an extraneous whitespace (go-gitea#24949) Show `bot` label next to username when rendering autor link if the user is a bot (go-gitea#24943) Improve some Forms (go-gitea#24878) Improve queue and logger context (go-gitea#24924) Fix ref type error (go-gitea#24941)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
gitea.com experienced the corrupted LevelQueue bug again.
I think the problem is clear now: if the keys in LevelDB went out-of-sync, the LevelQueue itself doesn't have the ability to recover, eg:
So it needs to dive into the LevelDB to remove all keys to recover the corrupted LevelQueue.
More comments are in TestCorruptedLevelQueue.