core, triedb/pathdb: final integration (snapshot integration pt 5) #30661
Conversation
rjl493456442
force-pushed
the
snapshot-integration-p5
branch
from
d249035 to
133900e
Compare
holiman
changed the title
rjl493456442
force-pushed
the
snapshot-integration-p5
branch
5 times, most recently
from
b879386 to
985a84e
Compare
| @@ -362,6 +430,7 @@ func (db *Database) Enable(root common.Hash) error { | |||
| // reset the persistent state id back to zero. | |||
| batch := db.diskdb.NewBatch() | |||
| rawdb.DeleteTrieJournal(batch) | |||
| rawdb.DeleteSnapshotRoot(batch) | |||
There was a problem hiding this comment.
Could you please explain why we need to delete the snapshot root here, I don't really understand
There was a problem hiding this comment.
This database entry indicates that the snapshot data on disk is associated with the specified state root.
The function Enable(root) means the trie node data on disk associates with the given state root, any leftover snapshot data should be purged and regenerated.
Deleting the SnapshotRoot entry signifies that all leftover storage data is considered obsolete.
| } | ||
| } | ||
| } | ||
| for addrHash, storages := range storageData { |
There was a problem hiding this comment.
I'm wondering whether it makes a difference in the speed of flushing if we sort the keys before trying to insert them, iirc some databases handle ordered key insertions better than random keys. This way we could also break on the first key that exceeds the genMarker
There was a problem hiding this comment.
The random insertion here won't significantly impact the performance, as the data receiver is a in-memory batch. But I would love to measure the performance difference later.
triedb/pathdb/generate.go
Outdated
| // into two parts. | ||
| func splitMarker(marker []byte) ([]byte, []byte) { | ||
| var accMarker []byte | ||
| if len(marker) > 0 { // []byte{} is the start, use nil for that |
There was a problem hiding this comment.
Do we need to check here that len(marker) >= HashLength?
There was a problem hiding this comment.
the generation marker could only be:
- []byte{}
- []byte{common.AddressLength}
- []byte{common.AddressLength + common.HashLength}
If the marker is corrupted, then the system could be very wrong and I don't mind panic here.
|
Took a first look now and it looks pretty good so far. I added a few questions/things I didn't understand while reviewing. The biggest changes (generator, generator_test, context) are modified copies from the snapshot package |
rjl493456442
force-pushed
the
snapshot-integration-p5
branch
from
985a84e to
9635f1d
Compare
rjl493456442
force-pushed
the
snapshot-integration-p5
branch
2 times, most recently
from
0245e41 to
c556e77
Compare
rjl493456442
force-pushed
the
snapshot-integration-p5
branch
from
cf0bbd6 to
87b17e7
Compare
rjl493456442
force-pushed
the
snapshot-integration-p5
branch
from
87b17e7 to
afecee4
Compare
rjl493456442
force-pushed
the
snapshot-integration-p5
branch
from
8feecf2 to
b4d7413
Compare
triedb/pathdb/disklayer.go
Outdated
| if dl.generator != nil { | ||
| dl.generator.stop() | ||
| progress = dl.generator.progressMarker() | ||
| log.Info("Terminated snapshot generation") |
fatal error: concurrent map iteration and map write
goroutine 3503687081 [running]:
internal/runtime/maps.fatal({0x1c0e410?, 0xc0d9bbe000?})
runtime/panic.go:1053 +0x18
internal/runtime/maps.(*Iter).Next(0xc003b47280?)
internal/runtime/maps/table.go:683 +0x86
github.com/ethereum/go-ethereum/triedb/pathdb.(*stateSet).accountList.Keys[...].func1()
maps/iter.go:27 +0x71
slices.AppendSeq[...](...)
slices/iter.go:50
slices.Collect[...](...)
slices/iter.go:58
slices.SortedFunc[...](0xc003b473d0, 0x1f96be0)
slices/iter.go:72 +0xd0
github.com/ethereum/go-ethereum/triedb/pathdb.(*stateSet).accountList(0xc003d26120)
github.com/ethereum/go-ethereum/triedb/pathdb/states.go:179 +0x13a
github.com/ethereum/go-ethereum/triedb/pathdb.newDiffAccountIterator({0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
…thereum#30661) In this pull request, snapshot generation in pathdb has been ported from the legacy state snapshot implementation. Additionally, when running in path mode, legacy state snapshot data is now managed by the pathdb based snapshot logic. Note: Existing snapshot data will be re-generated, regardless of whether it was previously fully constructed.
In this pull request, snapshot generation in the pathdb is ported from the legacy
state snapshot. Additionally, in path mode, the legacy state snapshot is now
handled by the pathdb-based snapshot implementation.
Note: the existing snapshot data will be re-generated regardless of it was fully constructed or not