Skip to content

Commit c12a16f

Browse files
authored
cmd: refactor dev/CMakeLists.txt (#2858)
1 parent 10ee6e9 commit c12a16f

File tree

11 files changed

+127
-209
lines changed

11 files changed

+127
-209
lines changed

cmake/run_smoke_tests.cmake

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,9 @@ endif()
1313
file(
1414
GLOB_RECURSE COMMANDS
1515
LIST_DIRECTORIES false
16-
"${SILKWORM_BUILD_DIR}/cmd/*${CMAKE_EXECUTABLE_SUFFIX}"
16+
"${SILKWORM_BUILD_DIR}/*${CMAKE_EXECUTABLE_SUFFIX}"
1717
)
18+
list(FILTER COMMANDS INCLUDE REGEX "(cli|cmd)/")
1819
if(NOT CMAKE_EXECUTABLE_SUFFIX)
1920
list(FILTER COMMANDS EXCLUDE REGEX "\\.")
2021
endif()

cmd/CMakeLists.txt

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,6 @@ if(NOT SILKWORM_CORE_ONLY)
1515
find_package(mimalloc REQUIRED)
1616
endif()
1717

18-
# Development utility tools
19-
add_subdirectory(dev)
20-
21-
# Silkworm runnable components
22-
2318
# [=] "all-in-one" Silkworm component
2419
# cmake-format: off
2520
set(SILKWORM_LIBRARIES

cmd/dev/CMakeLists.txt

Lines changed: 0 additions & 84 deletions
This file was deleted.

docs/CONTRIBUTING.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ If you need to update the list of builtin snapshots in Silkworm, the following p
9595
* update `erigon-snapshot` submodule to the new commit
9696
* generate the embedded C++ code bindings for predefined snapshots by executing from project home folder:
9797
```
98-
<build_folder>/cmd/dev/embed_toml -i third_party/erigon-snapshot -o silkworm/db/datastore/snapshots/config/chains
98+
<build_folder>/silkworm/dev/cli/embed_toml -i third_party/erigon-snapshot -o silkworm/db/datastore/snapshots/config/chains
9999
```
100100

101101

@@ -109,7 +109,7 @@ If you need to expand or modify the network configurations used by Silkworm, the
109109
1. add new or edit existing JSON genesis files in `silkworm/core/chain` following the naming convention `genesis_<network>.json`
110110
2. generate the C++ code bindings for JSON genesis files by executing from project home folder:
111111
```
112-
<build_folder>/cmd/dev/embed_json -i silkworm/core/chain -o silkworm/core/chain -w
112+
<build_folder>/silkworm/dev/cli/embed_json -i silkworm/core/chain -o silkworm/core/chain -w
113113
```
114114

115115

@@ -126,7 +126,7 @@ If you need to update the official specification imported by Silkworm, the follo
126126
3. copy and rename the generated JSON specification file into `silkworm/rpc/json_rpc/specification.json`, resolving the conflicts that may arise
127127
4. generate the C++ code bindings for JSON specification by executing from project home folder:
128128
```
129-
<build_folder>/cmd/dev/embed_json -i silkworm/rpc/json_rpc -o silkworm/rpc/json_rpc -p specification -n silkworm::rpc::json_rpc -w
129+
<build_folder>/silkworm/dev/cli/embed_json -i silkworm/rpc/json_rpc -o silkworm/rpc/json_rpc -p specification -n silkworm::rpc::json_rpc -w
130130
```
131131

132132
### Patch Local Specification

docs/db_toolbox.md

Lines changed: 40 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,37 @@
1-
Silkworm implements a small utility named `db_toolbox` which helps dealing with large LMDB datafile produced by Turbo-Geth.
2-
Simple usage is `./db_toolbox [OPTIONS] [SUBCOMMAND] [SUBCOMMANDOPTIONS]`
1+
# db_toolbox
32

4-
# OPTIONS
3+
Silkworm keeps recent chain data in MDBX database for faster access.
4+
The `db_toolbox` tool is a collection of utilities to perform operations on Silkworm MDBX database.
5+
6+
Simple usage is `db_toolbox [OPTIONS] [SUBCOMMAND] [SUBCOMMANDOPTIONS]`
7+
8+
## Examples
9+
10+
Dump the database table layout and stats
11+
12+
```
13+
db_toolbox --datadir ~/Library/Silkworm/ tables
14+
```
15+
16+
Dump the progress of sync stages (i.e. content of SyncStage table)
17+
18+
```
19+
db_toolbox --datadir ~/Library/Silkworm/ stages
20+
```
21+
22+
Clear content (i.e. delete all the rows) in LogAddressIndex and LogTopicIndex tables
23+
24+
```
25+
db_toolbox --datadir ~/Library/Silkworm/ --exclusive clear --names LogAddressIndex LogTopicIndex
26+
```
27+
28+
Reset the LogIndex stage progress to zero
29+
30+
```
31+
db_toolbox --datadir ~/Library/Silkworm/ --exclusive stage-set --name LogIndex --block 0
32+
```
33+
34+
## OPTIONS
535
Common options always specify the base data file to open :
636
- `--datadir` which indicates the **directory** path where `data.mdb` is located
737
- `--lmdb.mapSize` which indicates the **LMDB map size** of data.mdb
@@ -14,7 +44,7 @@ Omitting the specification of --lmdb.mapSize is allowed as long as the data file
1444
**Warning** : although db_toolbox protects against errors is highly discouraged to provide a value for --lmdb.mapSize lower than actual file size cause, as observed behavior, the result is a truncation of data file to a size matching --lmdb.mapSize thus causing the invalidation of all mappings for existing data.
1545

1646
# Subcommand : tables
17-
Usage `./db_toolbox --datadir <parent-directory-to-data.mdb> tables`
47+
Usage `db_toolbox --datadir <parent-directory-to-data.mdb> tables`
1848

1949
This subcommand requires no additional arguments and provides a detailed list of tables stored into data.mdb.
2050
Here is a sample output:
@@ -85,7 +115,7 @@ Each table reports:
85115
The bottom part of the report depicts the storage status of the data file.
86116

87117
# Subcommand : freelist
88-
Usage `./db_toolbox --datadir <parent-directory-to-data.mdb> freelist [--detail]`
118+
Usage `db_toolbox --datadir <parent-directory-to-data.mdb> freelist [--detail]`
89119

90120
This produces as output the sum of reclaimable space held in FREE_DBI.
91121
Sample :
@@ -142,25 +172,25 @@ Sample :
142172
```
143173

144174
# Subcommand : clear
145-
Usage `./db_toolbox --datadir <parent-directory-to-data.mdb> clear --names <list-of-table-names> [--drop]`
175+
Usage `db_toolbox --datadir <parent-directory-to-data.mdb> clear --names <list-of-table-names> [--drop]`
146176

147177
This command provides a handy way to empty a table from all records or drop it.
148178

149179
Example :
150-
`./db_toolbox --datadir <parent-directory-to-data.mdb> clear --names h b`
180+
`db_toolbox --datadir <parent-directory-to-data.mdb> clear --names h b`
151181

152182
will delete all records from tables `h` and `b` but the table (meant as a container) will remain into database.
153183

154184
Example :
155-
`./db_toolbox --datadir <parent-directory-to-data.mdb> clear --names h b --drop`
185+
`db_toolbox --datadir <parent-directory-to-data.mdb> clear --names h b --drop`
156186

157187
will delete tables `h` and `b` from database just like a SQL `drop` statement.
158188

159189
## Caveat
160190
Like all operations on LMDB the deletion of records (or of an entire table) lives within a writable transaction and by consequence requires database file to have enough space available to record all data pages which will be freed by the transaction. This implies the size of database file may grow.
161191

162192
# Subcommand : compact
163-
Usage `./db_toolbox --datadir <parent-directory-to-data.mdb> compact --workdir <parent-directory-to-compacted-data.mdb> [--replace] [--nobak]`
193+
Usage `db_toolbox --datadir <parent-directory-to-data.mdb> compact --workdir <parent-directory-to-compacted-data.mdb> [--replace] [--nobak]`
164194

165195
The purpose of this subcommand is to obtain a _compacted_ data file. The compaction process renumbers all data pages while reclaiming those previously freed by preceding transactions. This command is the implementation of `mdb_env_copy2` LMDB API call with `MDB_CP_COMPACT` flag.
166196
Running this command reports no progress and, ad indicative figure, took more than 6 hours to compact an 730GB data file on Windows with NMVe storage support.
@@ -290,7 +320,7 @@ This tools gives the user the ability to copy individual table(s) from one datab
290320

291321
Usage
292322
```
293-
./db_toolbox --datadir <parent-directory-to-source-data.mdb> copy --targetdir <parent-directory-to-target-data.mdb> \
323+
db_toolbox --datadir <parent-directory-to-source-data.mdb> copy --targetdir <parent-directory-to-target-data.mdb> \
294324
[--create --new.mapSize <value>] [--tables <list-of-table-names-to-copy>] \
295325
[--noempty] [--upsert] [--commit]
296326
```

cmd/dev/README.md renamed to docs/dev_tools.md

Lines changed: 18 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -18,25 +18,25 @@ The `check_log_indices` tool is a command-line utility to check the consistency
1818
Check only log address index from block 0 up to block 2'000'000
1919

2020
```
21-
cmd/dev/check_log_indices --to 2000000 --index address
21+
check_log_indices --to 2000000 --index address
2222
```
2323

2424
Check only log topic index for block 2'000'000
2525

2626
```
27-
cmd/dev/check_log_indices --from 2000000 --to 2000000 --index topic
27+
check_log_indices --from 2000000 --to 2000000 --index topic
2828
```
2929

3030
Check both log address and topic indices for block 17'500'000
3131

3232
```
33-
cmd/dev/check_log_indices --from 17500000 --to 17500000
33+
check_log_indices --from 17500000 --to 17500000
3434
```
3535

3636
Check both log address and topic indices from block 17'000'000 up to the tip (beware: long-running)
3737

3838
```
39-
cmd/dev/check_log_indices --from 17000000
39+
check_log_indices --from 17000000
4040
```
4141

4242
## Snapshots
@@ -61,7 +61,7 @@ The `snapshots` tool is a collection of small utilities and benchmark tests for
6161

6262
```
6363
Snapshots toolbox
64-
Usage: cmd/snapshots [OPTIONS]
64+
Usage: snapshots [OPTIONS]
6565
6666
Options:
6767
-h,--help Print this help message and exit
@@ -114,94 +114,56 @@ Currently available tools are:
114114
Download one snapshot from its magnet link and put it in torrent folder:
115115

116116
```
117-
cmd/dev/snapshots --tool download --magnet "magnet:?xt=urn:btih:83112dec4bec180cff67e01d6345c88c3134fd26&dn=v1-014500-015000-transactions.seg&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2f9.rarbg.com%3a2810%2fannounce&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a6969%2fannounce&tr=http%3a%2f%2ftracker.openbittorrent.com%3a80%2fannounce&tr=https%3a%2f%2fopentracker.i2p.rocks%3a443%2fannounce&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2ftracker.torrent.eu.org%3a451%2fannounce&tr=udp%3a%2f%2ftracker.tiny-vps.com%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.pomf.se%3a80%2fannounce&tr=udp%3a%2f%2ftracker.dler.org%3a6969%2fannounce&tr=udp%3a%2f%2fopen.demonii.com%3a1337%2fannounce&tr=udp%3a%2f%2fexplodie.org%3a6969%2fannounce&tr=udp%3a%2f%2fexodus.desync.com%3a6969%2fannounce&tr=https%3a%2f%2ftracker.nanoha.org%3a443%2fannounce&tr=https%3a%2f%2ftracker.lilithraws.org%3a443%2fannounce&tr=https%3a%2f%2ftr.burnabyhighstar.com%3a443%2fannounce&tr=http%3a%2f%2ftracker.mywaifu.best%3a6969%2fannounce&tr=http%3a%2f%2fbt.okmp3.ru%3a2710%2fannounce&tr=udp%3a%2f%2fzecircle.xyz%3a6969%2fannounce&tr=udp%3a%2f%2fyahor.ftp.sh%3a6969%2fannounce"
117+
snapshots --tool download --magnet "magnet:?xt=urn:btih:83112dec4bec180cff67e01d6345c88c3134fd26&dn=v1-014500-015000-transactions.seg&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2f9.rarbg.com%3a2810%2fannounce&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a6969%2fannounce&tr=http%3a%2f%2ftracker.openbittorrent.com%3a80%2fannounce&tr=https%3a%2f%2fopentracker.i2p.rocks%3a443%2fannounce&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2ftracker.torrent.eu.org%3a451%2fannounce&tr=udp%3a%2f%2ftracker.tiny-vps.com%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.pomf.se%3a80%2fannounce&tr=udp%3a%2f%2ftracker.dler.org%3a6969%2fannounce&tr=udp%3a%2f%2fopen.demonii.com%3a1337%2fannounce&tr=udp%3a%2f%2fexplodie.org%3a6969%2fannounce&tr=udp%3a%2f%2fexodus.desync.com%3a6969%2fannounce&tr=https%3a%2f%2ftracker.nanoha.org%3a443%2fannounce&tr=https%3a%2f%2ftracker.lilithraws.org%3a443%2fannounce&tr=https%3a%2f%2ftr.burnabyhighstar.com%3a443%2fannounce&tr=http%3a%2f%2ftracker.mywaifu.best%3a6969%2fannounce&tr=http%3a%2f%2fbt.okmp3.ru%3a2710%2fannounce&tr=udp%3a%2f%2fzecircle.xyz%3a6969%2fannounce&tr=udp%3a%2f%2fyahor.ftp.sh%3a6969%2fannounce"
118118
```
119119

120120
Download all snapshots from the magnet links contained in magnet file and put them in torrent folder:
121121

122122
```
123-
cmd/dev/snapshots --tool download --magnet_file .magnet_links --log.verbosity debug --active_downloads 3
123+
snapshots --tool download --magnet_file .magnet_links --log.verbosity debug --active_downloads 3
124124
```
125125

126126
Count how many block headers are present in header snapshots under torrent folder:
127127

128128
```
129-
cmd/dev/snapshots --tool count_headers --repetitions 1 --log.verbosity info
129+
snapshots --tool count_headers --repetitions 1 --log.verbosity info
130130
```
131131

132132
Create indexes for target snapshot under torrent folder:
133133

134134
```
135-
cmd/dev/snapshots --tool create_index --file v1-000000-000500-headers.seg --log.verbosity info
135+
snapshots --tool create_index --file v1-000000-000500-headers.seg --log.verbosity info
136136
```
137137

138138
Search block header by number in one snapshot
139139

140140
```
141-
cmd/dev/snapshots --tool lookup_header --snapshot_file v1-001500-002000-headers.seg --number 1500013
141+
snapshots --tool lookup_header --snapshot_file v1-001500-002000-headers.seg --number 1500013
142142
```
143143

144144
Search block body by number in all snapshots
145145

146146
```
147-
cmd/dev/snapshots --tool lookup_body --number 1500012
147+
snapshots --tool lookup_body --number 1500012
148148
```
149149

150150
Search block body by number in one snapshot
151151

152152
```
153-
cmd/dev/snapshots --tool lookup_body --snapshot_file v1-001500-002000-bodies.seg --number 1500012
153+
snapshots --tool lookup_body --snapshot_file v1-001500-002000-bodies.seg --number 1500012
154154
```
155155

156156
Search transaction by hash in all snapshots
157157

158158
```
159-
cmd/dev/snapshots --tool lookup_txn --hash 0x3ba9a1f95b96d0a43093b1ade1174133ea88ca395e60fe9fd8144098ff7a441f
159+
snapshots --tool lookup_txn --hash 0x3ba9a1f95b96d0a43093b1ade1174133ea88ca395e60fe9fd8144098ff7a441f
160160
```
161161

162162
Search transaction by hash or by progressive identifier in one snapshot
163163

164164
```
165-
cmd/dev/snapshots --tool lookup_txn --snapshot_file v1-001500-002000-transactions.seg --hash 0x3ba9a1f95b96d0a43093b1ade1174133ea88ca395e60fe9fd8144098ff7a441f
166-
cmd/dev/snapshots --tool lookup_txn --snapshot_file v1-001500-002000-transactions.seg --number 7341272
167-
```
168-
169-
## Database Toolbox
170-
171-
### Overview
172-
173-
Silkworm keeps recent chain data in MDBX database for faster access.
174-
175-
### The `db_toolbox` tool
176-
177-
The `db_toolbox` tool is a collection of utilities to perform operations on Silkworm MDBX database.
178-
179-
#### Synopsis
180-
181-
#### Examples
182-
183-
Dump the database table layout and stats
184-
185-
```
186-
cmd/dev/db_toolbox --datadir ~/Library/Silkworm/ tables
187-
```
188-
189-
Dump the progress of sync stages (i.e. content of SyncStage table)
190-
191-
```
192-
cmd/dev/db_toolbox --datadir ~/Library/Silkworm/ stages
193-
```
194-
195-
Clear content (i.e. delete all the rows) in LogAddressIndex and LogTopicIndex tables
196-
197-
```
198-
cmd/dev/db_toolbox --datadir ~/Library/Silkworm/ --exclusive clear --names LogAddressIndex LogTopicIndex
199-
```
200-
201-
Reset the LogIndex stage progress to zero
202-
203-
```
204-
cmd/dev/db_toolbox --datadir ~/Library/Silkworm/ --exclusive stage-set --name LogIndex --block 0
165+
snapshots --tool lookup_txn --snapshot_file v1-001500-002000-transactions.seg --hash 0x3ba9a1f95b96d0a43093b1ade1174133ea88ca395e60fe9fd8144098ff7a441f
166+
snapshots --tool lookup_txn --snapshot_file v1-001500-002000-transactions.seg --number 7341272
205167
```
206168

207169
## gRPC Toolbox
@@ -221,18 +183,18 @@ The `db_toolbox` tool is a collection of utilities to query the KV/ETHBACKEND gR
221183
Print the number of timestamps in which the specified account has changed state
222184

223185
```
224-
cmd/dev/grpc_toolbox kv_index_range --table AccountsHistoryIdx --key 0x616a3E55a20dD54CC9fBb63D8333D89c275c9D90
186+
grpc_toolbox kv_index_range --table AccountsHistoryIdx --key 0x616a3E55a20dD54CC9fBb63D8333D89c275c9D90
225187
```
226188

227189
Print the first 10 changes in account state history using verbose mode (i.e. print keys and values)
228190

229191
```
230-
cmd/dev/grpc_toolbox kv_history_range --table AccountsHistory --limit 10 --verbose
192+
grpc_toolbox kv_history_range --table AccountsHistory --limit 10 --verbose
231193
```
232194

233195
Print the first 10 changes in account state for the specified key range using verbose mode (i.e. print keys and values)
234196

235197
```
236-
cmd/dev/grpc_toolbox kv_domain_range --table accounts --from_key 0x616a3E55a20dD54CC9fBb63D8333D89c275c9D90 \
198+
grpc_toolbox kv_domain_range --table accounts --from_key 0x616a3E55a20dD54CC9fBb63D8333D89c275c9D90 \
237199
--to_key 0x716a3E55a20dD54CC9fBb63D8333D89c275c9D90 --timestamp 100000000 --limit 10 --verbose
238200
```

0 commit comments

Comments
 (0)