Skip to content

Deprecate heapsize #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jun 11, 2019
Merged

Deprecate heapsize #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
c2e19fc
use parity-util-mem to get heapsize of memory db
cheme Feb 22, 2019
4284b85
Use github temp instead of local
cheme Feb 27, 2019
cba8e5e
keep version update for later (first need to get util-mem published)
cheme Feb 27, 2019
2a1d3bb
Deprecated heapsize only if feature activated (note the value was alr…
cheme Feb 27, 2019
7d8022c
Merge branch 'master' into deprecate_heapsize
cheme May 31, 2019
0112297
improve import of malloc, and change deprecated message.
cheme May 31, 2019
6c01e71
Use published crate
cheme May 31, 2019
ecef806
Full crate version update (mem db is use by hash and trie so updating
cheme Jun 3, 2019
4fbc110
no_std for parity util mem
cheme Jun 3, 2019
cfe6c0b
Add prefix and empty prefix.
cheme Jun 6, 2019
e670d38
Needed clone on kf
cheme Jun 6, 2019
2ec922a
manually implement partialeq to memorydb
cheme Jun 6, 2019
3867002
Fix maybedebug trait absence.
cheme Jun 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hash-db/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "hash-db"
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
description = "Trait for hash-keyed databases."
license = "Apache-2.0"
Expand Down
7 changes: 7 additions & 0 deletions hash-db/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ pub trait MaybeDebug {}
#[cfg(not(feature = "std"))]
impl<T> MaybeDebug for T {}


/// Empty prefix constant.
pub static EMPTY_PREFIX: Prefix<'static> = &[];

/// Prefix byte information for avoding rc.
pub type Prefix<'a> = &'a[u8];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you help me understand what this does?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is an additional information that can be use to ensure key uniqueness in the hashdb.
Basically we do not reference count the key value; if some different nodes in the trie contains the same content a pruning conflict will occur.
For ethereum, the constructed keys are long and unique enough so that leaf contains a part of the key big enough to ensure that it is hash is unique and no rc is needed (intermediatory node containing those leaf hash that is also the case). This really depends on the way keys are build.

In substrate where the use cases are more open, there was this pruning issue, solution there was to add the path in the trie to the info (unique by nature of the trie), we did not have to use reference count.
There trie issue with the merged pr contains more info (I did not understand directly so I asked my questions there).
So for parity ethereum, this is strictly overhead, note that with my trie pr about removal of extension, this overhead gets a bit reduced (there is also an interesting scheme that add the prefix in an ordered encoding to try to allow iteration directly from rocksdb but I do not know if it is worth the overhead (unless the state is totally pruned that does not allow state iteration still can be use for some analysis)).


/// Trait describing an object that can hash a slice of bytes. Used to abstract
/// other types over the hashing algorithm. Defines a single `hash` method and an
/// `Out` associated type with the necessary bounds.
Expand Down
2 changes: 1 addition & 1 deletion hash256-std-hasher/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[package]
name = "hash256-std-hasher"
description = "Standard library hasher for 256-bit prehashed keys."
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
license = "Apache-2.0"
homepage = "https://github.com/paritytech/trie"
Expand Down
12 changes: 7 additions & 5 deletions memory-db/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,29 @@
[package]
name = "memory-db"
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
description = "In-memory implementation of hash-db, useful for tests"
repository = "https://github.com/paritytech/trie"
license = "Apache-2.0"

[dependencies]
heapsize = { version = "0.4", optional = true}
hash-db = { path = "../hash-db", default-features = false, version = "0.12.2"}
heapsize = { version = "0.4", optional = true }
parity-util-mem = { version = "0.1", default-features = false }
hash-db = { path = "../hash-db", default-features = false, version = "0.12.3"}
hashmap_core = { version = "0.1" }

[dev-dependencies]
keccak-hasher = { path = "../test-support/keccak-hasher", version = "0.12.2"}
keccak-hasher = { path = "../test-support/keccak-hasher", version = "0.12.3"}
criterion = "0.2.8"

[features]
default = ["std"]
std = [
"heapsize",
"hash-db/std",
"hashmap_core/disable",
"parity-util-mem/std",
]
deprecated = [ "heapsize" ]

[[bench]]
name = "bench"
Expand Down
65 changes: 62 additions & 3 deletions memory-db/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
#![cfg_attr(not(feature = "std"), feature(alloc))]

extern crate hash_db;
extern crate parity_util_mem;
#[cfg(feature = "deprecated")]
#[cfg(feature = "std")]
extern crate heapsize;
#[cfg(not(feature = "std"))]
Expand All @@ -27,6 +29,8 @@ extern crate alloc;
#[cfg(test)] extern crate keccak_hasher;

use hash_db::{HashDB, HashDBRef, PlainDB, PlainDBRef, Hasher as KeyHasher, AsHashDB, AsPlainDB};
use parity_util_mem::{MallocSizeOf, MallocSizeOfOps};
#[cfg(feature = "deprecated")]
#[cfg(feature = "std")]
use heapsize::HeapSizeOf;
#[cfg(feature = "std")]
Expand Down Expand Up @@ -58,6 +62,15 @@ use core::{
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;

#[cfg(feature = "std")]
pub trait MaybeDebug: std::fmt::Debug {}
#[cfg(feature = "std")]
impl<T: std::fmt::Debug> MaybeDebug for T {}
#[cfg(not(feature = "std"))]
pub trait MaybeDebug {}
#[cfg(not(feature = "std"))]
impl<T> MaybeDebug for T {}

/// Reference-counted memory-based `HashDB` implementation.
///
/// Use `new()` to create a new database. Insert items with `insert()`, remove items
Expand Down Expand Up @@ -105,7 +118,7 @@ use alloc::vec::Vec;
/// assert!(!m.contains(&k, &[]));
/// }
/// ```
#[derive(Clone, PartialEq)]
#[derive(Clone)]
pub struct MemoryDB<H, KF, T>
where
H: KeyHasher,
Expand All @@ -117,13 +130,39 @@ pub struct MemoryDB<H, KF, T>
_kf: PhantomData<KF>,
}

impl<H, KF, T> PartialEq<MemoryDB<H, KF, T>> for MemoryDB<H, KF, T>
where
H: KeyHasher,
KF: KeyFunction<H>,
<KF as KeyFunction<H>>::Key: Eq + MaybeDebug,
T: Eq + MaybeDebug,
{
fn eq(&self, other: &MemoryDB<H, KF, T>) -> bool {
for a in self.data.iter() {
match other.data.get(&a.0) {
Some(v) if v != a.1 => return false,
None => return false,
_ => (),
}
}
true
}
}

impl<H, KF, T> Eq for MemoryDB<H, KF, T>
where
H: KeyHasher,
KF: KeyFunction<H>,
<KF as KeyFunction<H>>::Key: Eq + MaybeDebug,
T: Eq + MaybeDebug,
{}

pub trait KeyFunction<H: KeyHasher> {
type Key: Send + Sync + Clone + hash::Hash + Eq ;
type Key: Send + Sync + Clone + hash::Hash + Eq;

fn key(hash: &H::Out, prefix: &[u8]) -> Self::Key;
}


/// Make database key from hash and prefix.
pub fn prefixed_key<H: KeyHasher>(key: &H::Out, prefix: &[u8]) -> Vec<u8> {
let mut prefixed_key = Vec::with_capacity(key.as_ref().len() + prefix.len());
Expand All @@ -137,6 +176,7 @@ pub fn hash_key<H: KeyHasher>(key: &H::Out, _prefix: &[u8]) -> H::Out {
key.clone()
}

#[derive(Clone,Debug)]
/// Key function that only uses the hash
pub struct HashKey<H: KeyHasher>(PhantomData<H>);

Expand All @@ -148,6 +188,7 @@ impl<H: KeyHasher> KeyFunction<H> for HashKey<H> {
}
}

#[derive(Clone,Debug)]
/// Key function that concatenates prefix and hash.
pub struct PrefixedKey<H: KeyHasher>(PhantomData<H>);

Expand Down Expand Up @@ -297,20 +338,38 @@ where
}
}

#[cfg(feature = "deprecated")]
#[cfg(feature = "std")]
impl<H, KF, T> MemoryDB<H, KF, T>
where
H: KeyHasher,
T: HeapSizeOf,
KF: KeyFunction<H>,
{
#[deprecated(since="0.12.0", note="please use `size_of` instead")]
/// Returns the size of allocated heap memory
pub fn mem_used(&self) -> usize {
0//self.data.heap_size_of_children()
// TODO Reenable above when HeapSizeOf supports arrays.
}
}

impl<H, KF, T> MallocSizeOf for MemoryDB<H, KF, T>
where
H: KeyHasher,
H::Out: MallocSizeOf,
T: MallocSizeOf,
KF: KeyFunction<H>,
KF::Key: MallocSizeOf,
{
fn size_of(&self, ops: &mut MallocSizeOfOps) -> usize {
self.data.size_of(ops)
+ self.null_node_data.size_of(ops)
+ self.hashed_null_node.size_of(ops)
}
}


impl<H, KF, T> PlainDB<H::Out, T> for MemoryDB<H, KF, T>
where
H: KeyHasher,
Expand Down
6 changes: 3 additions & 3 deletions test-support/keccak-hasher/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
[package]
name = "keccak-hasher"
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
description = "Keccak-256 implementation of the Hasher trait"
repository = "https://github.com/paritytech/parity/"
license = "Apache-2.0"

[dependencies]
tiny-keccak = "1.4.2"
hash-db = { path = "../../hash-db", default-features = false, version = "0.12.2" }
hash256-std-hasher = { path = "../../hash256-std-hasher", version = "0.12.2" }
hash-db = { path = "../../hash-db", default-features = false, version = "0.12.3" }
hash256-std-hasher = { path = "../../hash256-std-hasher", version = "0.12.3" }

[features]
default = ["std"]
Expand Down
14 changes: 7 additions & 7 deletions test-support/reference-trie/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
[package]
name = "reference-trie"
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
description = "Simple reference trie format"
repository = "https://github.com/paritytech/trie/"
license = "Apache-2.0"

[dependencies]
hash-db = { path = "../../hash-db" , version = "0.12.2"}
hash256-std-hasher = { path = "../../hash256-std-hasher", version = "0.12.2" }
keccak-hasher = { path = "../keccak-hasher", version = "0.12.2" }
trie-db = { path = "../../trie-db", default-features = false, version = "0.12.2"}
trie-root = { path = "../../trie-root", default-features = false, version = "0.12.2" }
hash-db = { path = "../../hash-db" , version = "0.12.3"}
hash256-std-hasher = { path = "../../hash256-std-hasher", version = "0.12.3" }
keccak-hasher = { path = "../keccak-hasher", version = "0.12.3" }
trie-db = { path = "../../trie-db", default-features = false, version = "0.12.3"}
trie-root = { path = "../../trie-root", default-features = false, version = "0.12.3" }
parity-codec = "3.0"
parity-codec-derive = "3.0"

[dev-dependencies]
trie-bench = { path = "../trie-bench", version = "0.12.2" }
trie-bench = { path = "../trie-bench", version = "0.12.3" }
criterion = "0.2.8"

[[bench]]
Expand Down
14 changes: 7 additions & 7 deletions test-support/trie-bench/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
[package]
name = "trie-bench"
description = "Standard benchmarking suite for tries"
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
license = "Apache-2.0"

[dependencies]
keccak-hasher = { path = "../keccak-hasher", version = "0.12.2" }
trie-standardmap = { path = "../trie-standardmap", version = "0.12.2" }
hash-db = { path = "../../hash-db" , version = "0.12.2"}
memory-db = { path = "../../memory-db", version = "0.12.2" }
trie-root = { path = "../../trie-root", version = "0.12.2" }
trie-db = { path = "../../trie-db", version = "0.12.2" }
keccak-hasher = { path = "../keccak-hasher", version = "0.12.3" }
trie-standardmap = { path = "../trie-standardmap", version = "0.12.3" }
hash-db = { path = "../../hash-db" , version = "0.12.3"}
memory-db = { path = "../../memory-db", version = "0.12.3" }
trie-root = { path = "../../trie-root", version = "0.12.3" }
trie-db = { path = "../../trie-db", version = "0.12.3" }
criterion = "0.2.8"
parity-codec = "3.0"
4 changes: 2 additions & 2 deletions test-support/trie-standardmap/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ authors = ["Parity Technologies <[email protected]>"]
license = "Apache-2.0"

[dependencies]
keccak-hasher = { path = "../keccak-hasher", version = "0.12.2"}
hash-db = { path = "../../hash-db" , version = "0.12.2"}
keccak-hasher = { path = "../keccak-hasher", version = "0.12.3"}
hash-db = { path = "../../hash-db" , version = "0.12.3"}
14 changes: 7 additions & 7 deletions trie-db/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "trie-db"
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
description = "Merkle-Patricia Trie generic over key hasher and node encoding"
repository = "https://github.com/paritytech/trie"
Expand All @@ -10,17 +10,17 @@ license = "Apache-2.0"
log = "0.4"
rand = { version = "0.6", default-features = false }
elastic-array = { version = "0.10", default-features = false }
hash-db = { path = "../hash-db", default-features = false, version = "0.12.2"}
hash-db = { path = "../hash-db", default-features = false, version = "0.12.3"}
hashmap_core = { version = "0.1" }

[dev-dependencies]
env_logger = "0.6"
memory-db = { path = "../memory-db", version = "0.12.2" }
trie-root = { path = "../trie-root", version = "0.12.2"}
trie-standardmap = { path = "../test-support/trie-standardmap", version = "0.12.2" }
keccak-hasher = { path = "../test-support/keccak-hasher", version = "0.12.2" }
memory-db = { path = "../memory-db", version = "0.12.3" }
trie-root = { path = "../trie-root", version = "0.12.3"}
trie-standardmap = { path = "../test-support/trie-standardmap", version = "0.12.3" }
keccak-hasher = { path = "../test-support/keccak-hasher", version = "0.12.3" }
# DISABLE the following line when publishing until cyclic dependencies are resolved https://github.com/rust-lang/cargo/issues/4242
reference-trie = { path = "../test-support/reference-trie", version = "0.12.2" }
reference-trie = { path = "../test-support/reference-trie", version = "0.12.3" }
hex-literal = "0.1"
criterion = "0.2.8"

Expand Down
10 changes: 5 additions & 5 deletions trie-root/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
[package]
name = "trie-root"
version = "0.12.2"
version = "0.12.3"
authors = ["Parity Technologies <[email protected]>"]
description = "In-memory patricia trie operations"
repository = "https://github.com/paritytech/trie"
license = "Apache-2.0"
categories = [ "no-std" ]

[dependencies]
hash-db = { path = "../hash-db", default-features = false, version = "0.12.2"}
hash-db = { path = "../hash-db", default-features = false, version = "0.12.3"}

[dev-dependencies]
hex-literal = "0.1"
keccak-hasher = { path = "../test-support/keccak-hasher", version = "0.12.2" }
trie-standardmap = { path = "../test-support/trie-standardmap", version = "0.12.2" }
keccak-hasher = { path = "../test-support/keccak-hasher", version = "0.12.3" }
trie-standardmap = { path = "../test-support/trie-standardmap", version = "0.12.3" }
# DISABLE the following line when publishing until cyclic dependencies are resolved https://github.com/rust-lang/cargo/issues/4242
reference-trie = { path = "../test-support/reference-trie", version = "0.12.2" }
reference-trie = { path = "../test-support/reference-trie", version = "0.12.3" }

[features]
default = ["std"]
Expand Down