[WIP] core, trie: refactor trie API #26995 #1147
Open
+6,730
−2,540
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* eth/downloader: refactor downloader + queue downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache downloader: more accurate deliverytime calculation, less mem overhead in state requests downloader/queue: increase underlying buffer of results, new throttle mechanism eth/downloader: updates to tests eth/downloader: fix up some review concerns eth/downloader/queue: minor fixes eth/downloader: minor fixes after review call eth/downloader: testcases for queue.go eth/downloader: minor change, don't set progress unless progress... eth/downloader: fix flaw which prevented useless peers from being dropped eth/downloader: try to fix tests eth/downloader: verify non-deliveries against advertised remote head eth/downloader: fix flaw with checking closed-status causing hang eth/downloader: hashing avoidance eth/downloader: review concerns + simplify resultcache and queue eth/downloader: add back some locks, address review concerns downloader/queue: fix remaining lock flaw * eth/downloader: nitpick fixes * eth/downloader: remove the *2*3/4 throttling threshold dance * eth/downloader: print correct throttle threshold in stats Co-authored-by: Péter Szilágyi <[email protected]>
This changes how the downloader works, a little bit. Previously, when block sync started, we immediately started filling up to 8192 blocks. Usually this is fine, blocks are small in the early numbers. The threshold then is lowered as we measure the size of the blocks that are filled. However, if the node is shut down and restarts syncing while we're in a heavy segment, that might be bad. This PR introduces a more conservative initial threshold of 2K blocks instead.
Fixes a special case when the trie only has a single trie node and the range proof only contains a single element.
…um#22667 * core/state/snapshot: reuse memory data instead of hitting disk when generating * trie: minor nitpicks wrt the resolver optimization * core/state/snapshot, trie: use key/value store for resolver * trie: fix linter Co-authored-by: Péter Szilágyi <[email protected]>
…eum#22760 * trie: add benchmark for proofless range * trie: remove unused returns + use stacktrie
…m#23415 Some tests take quite some time during exit, which I think causes some appveyor fails like this: https://ci.appveyor.com/project/ethereum/go-ethereum/builds/39511210/job/xhom84eg2e4uulq3 One of the things that seem to take time during exit is waiting (up to 100ms) for the syncbloom to close. This PR changes it to use a channel, instead of looping with a 100ms wait. This also includes some unrelated changes improving the reliability of eth/fetcher tests, which fail a lot because they are time-dependent.
This functionality is needed in new path-based storage scheme, but can be implemented in a seperate PR though. When an account is deleted, then all the storage slots should be nuked out from the disk as well. In hash-based storage scheme they are still left in the disk but in new scheme, they will be iterated and marked as deleted. But why the NodeBlob API is needed in this scenario? Because when the node is marked deleted, the previous value is also required to be recorded to construct the reverse diff.
* trie: fix memory leak in trie iterator In the trie iterator, live nodes are tracked in a stack while iterating. Popped node states should be explictly set to nil in order to get garbage-collected. * trie: fix empty trie iterator
Trie tracer is an auxiliary tool to capture all deleted nodes which can't be captured by trie.Committer. The deleted nodes can be removed from the disk later.
* core, trie, eth, cmd: rework preimage store * trie: address comment
* all: rework trie and trie committer * all: get rid of internal cache in trie * all: fixes * trie: polish * core, trie: address comments * trie: fix imports * core/state: address comments * core/state/snapshot: polish * trie: remove unused code * trie: update tests * trie: don't set db as nil * trie: address comments * trie: unskip test
…written ethereum#25458 * core: use TryGetAccount to read where TryUpdateAccount has been used to write * Gary's review feedback * implement Gary's suggestion * fix bug + rename NewSecure into NewStateTrie * trie: add backwards-compatibility aliases for SecureTrie * Update database.go * make the linter happy Co-authored-by: Felix Lange <[email protected]> Co-authored-by: rjl493456442 <[email protected]>
It's a trivial PR to hide the error log when the trie node is not found in the database. The idea for this change is for all TryXXX functions, the error is already returned and we don't need to fire a log explicitly. Recently there are a few tickets ethereum#25613 ethereum#25589 reporting that the trie nodes are missing because of debug.SetHead. The root cause is after resetting, the chain rewinds to a historical point and re-imports the blocks on top. Since the node is already synced and started to accept transactions previously, these transactions are still kept in the txpool and verified by txpool with a live state. This live state is constructed based on the live trie database, which is changed fast by node referencing and de-referencing. Unfortunately, when we construct a live state(like the state in txpool), we don't reference the state we have. The blockchain will garbage collect the intermediate version nodes in another thread which leads the broken live state. The best solution for this is to forcibly obtain a reference for all live states we create and call release function once it's used up. But it might end up with more junks persisted into disk. Will try to find an elegant solution later in the following PR.
This PR includes minor updates to comments in trie/committer that reference insertion to the db, and adds an err != nil check for the return value of preimages.commit.
Co-authored-by: Martin Holst Swende <[email protected]>
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon. Apart from that, a few changes are also included which is worth mentioning: - port the changes in the stacktrie, tracking the path prefix of nodes during commit - use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
This PR fixes an error in trie commit. If the trie.root is nil, it can be two possible scenarios: - The trie was empty, and no change happens - The trie was non-empty and all nodes are dropped For the latter one, we should collect the deletions and apply them into database(e.g. in PBSS).
* all: cleanup trie interface * eth, trie: address comments
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors. This also moves some code from trie package to rawdb package. This PR is intended to be a no-functionality-change prep PR for ethereum#25963 . --------- Co-authored-by: Gary Rong <[email protected]>
This changes the Trie interface to add the plain account address as a parameter to all storage-related methods. After the introduction of the TryAccount* functions, TryGet, TryUpdate and TryDelete are now only meant to read an account's storage. In their current form, they assume that an account storage is stored in a separate trie, and that the hashing of the slot is independent of its account's address. The proposed structure for a stateless storage breaks these two assumptions: the hashing of a slot key requires the address and all slots and accounts are stored in a single trie. This PR therefore adds an address parameter to the interface. It is ignored in the MPT version, so this change has no functional impact, however it will reduce the diff size when merging verkle trees.
This change renames StateTrie methods to remove the Try* prefix. We added the Trie methods with prefix 'Try' a long time ago, working around the problem that most existing methods of Trie did not return the database error. This weird naming convention has persisted until now. Co-authored-by: Gary Rong <[email protected]>
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned. The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Ref: ethereum#26995
Types of changes
What types of changes does your code introduce to XDC network?
Put an
✅in the boxes that applyImpacted Components
Which part of the codebase this PR will touch base on,
Put an
✅in the boxes that applyChecklist
Put an
✅in the boxes once you have confirmed below actions (or provide reasons on not doing so) that