Skip to content

Remove heapsize #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 49 commits into from
May 31, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
25270be
Mem tools by including trait and specific implementation in the crate
cheme Dec 6, 2018
94bb293
Plug malloc_size_of as a patched copy. Documentation is done by
cheme Dec 7, 2018
197a27d
A few additional impl for parity-eth use
cheme Dec 7, 2018
f816aaf
Removal of heapsize (no proper crate versioning and pending on mem crate
cheme Dec 7, 2018
2a0cc08
Allocators.
cheme Dec 7, 2018
418e03b
heapsize estimation for allocator without use_size fn.
cheme Dec 10, 2018
6105606
Init no_std compat.
cheme Dec 10, 2018
59b4d55
Remove conditianal test code.
cheme Dec 10, 2018
efadc31
Renaming of mem repo to crate name.
cheme Dec 11, 2018
b2f1708
Remove conditional mettering for now (will need some testing to check if
cheme Dec 11, 2018
fb127a7
tab
cheme Dec 11, 2018
d5aa899
Remove features get form malloc_size and simply delete code instead.
cheme Jan 2, 2019
fd9dc92
Update parity-util-mem/README.md
pepyakin Jan 2, 2019
3df427b
Missing windows import
cheme Jan 3, 2019
da88f97
Merge branch 'mem-tools2' of github.com:cheme/parity-common into mem-…
cheme Jan 3, 2019
ef9a5b6
Use self for winapi.
cheme Jan 3, 2019
1b0b2e3
Correct import.
cheme Jan 3, 2019
8839576
Merge branch 'master' into rem_heapsize
cheme Jan 3, 2019
5fd398f
Small renaming, test confirm that only memorydb need update before
cheme Jan 3, 2019
5340912
shallow size on Arc with malloc return -1 (max val), replacing with
cheme Jan 3, 2019
8ada814
use cfg_if to makes feature choice more explicit.
cheme Jan 7, 2019
e3516d2
Plug on github branch (test brokens)
cheme Jan 9, 2019
61c60b0
Remove servo specific impls (unused).
cheme Jan 9, 2019
767dc90
`estimate_heapsize` in travis.
cheme Jan 9, 2019
9876b65
Merge branch 'master' into mem-tools2
cheme Jan 9, 2019
a8a9b0e
Merge branch 'master' into mem-tools2
cheme Jan 9, 2019
a31851e
Remove last serde and remove servo_arc reference.
cheme Jan 10, 2019
73e8c7e
Add missing version
cheme Jan 10, 2019
64135ac
Use malloc_size_of_derive from crates.
cheme Jan 12, 2019
af24ca4
Merge branch 'mem-tools2' into rem_heapsize
cheme Jan 14, 2019
00a0931
Merge branch 'mem-tools2' into rem_heapsize
cheme Jan 14, 2019
d85752e
Update versioning for breaking change
cheme Jan 14, 2019
32beba3
Merge branch 'rem_heapsize' of github.com:cheme/parity-common into re…
cheme Jan 14, 2019
ac83cdd
Clean heapsize uses.
cheme Jan 14, 2019
bce898c
Updating uint version, 0.6.0 has been published already.
cheme Jan 16, 2019
821c02c
Missing reference to uint 0.7.0
cheme Jan 16, 2019
85597c0
Merge branch 'master' into rem_heapsize
cheme Jan 16, 2019
d770316
Merge branch 'master' into rem_heapsize, and update uint readme crate
cheme Jan 16, 2019
a0a2577
Merge branch 'master' into rem_heapsize
cheme Jan 16, 2019
ca27409
restore test dependency
cheme Jan 16, 2019
c6a6982
Merge branch 'master' into rem_heapsize, remove clear_on_drop from
cheme May 31, 2019
b7de962
Update keccak-hash/Cargo.toml
cheme May 31, 2019
a65dcdb
fix readme comment
cheme May 31, 2019
45860cd
Allow no_std for estimate_size
cheme May 31, 2019
ba4dd7c
Merge branch 'rem_heapsize' of github.com:cheme/parity-common into re…
cheme May 31, 2019
96c860b
use a version, not sure a major version only is valid for publishing
cheme May 31, 2019
1b0bc65
fix warnings
cheme May 31, 2019
0fbc1ce
fix warnings
cheme May 31, 2019
18d3f99
fix string import warning
cheme May 31, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions ethbloom/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "ethbloom"
version = "0.6.4"
version = "0.7.0"
authors = ["Parity Technologies <[email protected]>"]
description = "Ethereum bloom filter"
license = "MIT"
Expand All @@ -11,7 +11,7 @@ repository = "https://github.com/paritytech/parity-common"
[dependencies]
tiny-keccak = "1.4"
crunchy = { version = "0.2", default-features = false, features = ["limit_256"] }
fixed-hash = { path = "../fixed-hash", version = "0.3", default-features = false }
fixed-hash = { path = "../fixed-hash", version = "0.4", default-features = false }
impl-serde = { path = "../primitive-types/impls/serde", version = "0.2", default-features = false, optional = true }
impl-rlp = { path = "../primitive-types/impls/rlp", version = "0.2", default-features = false }

Expand All @@ -20,7 +20,6 @@ rand = { version = "0.4" }
hex-literal = "0.1.1"

[features]
default = ["std", "heapsize", "serialize", "fixed-hash/libc", "fixed-hash/rustc-hex"]
default = ["std", "serialize", "fixed-hash/libc", "fixed-hash/rustc-hex"]
std = ["fixed-hash/std", "crunchy/std"]
heapsize = ["fixed-hash/heapsize"]
serialize = ["std", "impl-serde"]
13 changes: 6 additions & 7 deletions ethereum-types/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,24 +1,23 @@
[package]
name = "ethereum-types"
version = "0.6.0"
version = "0.7.0"
authors = ["Parity Technologies <[email protected]>"]
license = "MIT"
homepage = "https://github.com/paritytech/parity-common"
description = "Ethereum types"

[dependencies]
ethbloom = { path = "../ethbloom", version = "0.6", default-features = false }
fixed-hash = { path = "../fixed-hash", version = "0.3", default-features = false, features = ["byteorder", "rustc-hex"] }
uint = { path = "../uint", version = "0.7", default-features = false }
primitive-types = { path = "../primitive-types", version = "0.3", features = ["rlp", "byteorder", "rustc-hex"], default-features = false }
ethbloom = { path = "../ethbloom", version = "0.7", default-features = false }
fixed-hash = { path = "../fixed-hash", version = "0.4", default-features = false, features = ["byteorder", "rustc-hex"] }
uint = { path = "../uint", version = "0.8", default-features = false }
primitive-types = { path = "../primitive-types", version = "0.4", features = ["rlp", "byteorder", "rustc-hex"], default-features = false }
impl-serde = { path = "../primitive-types/impls/serde", version = "0.2", default-features = false, optional = true }
impl-rlp = { path = "../primitive-types/impls/rlp", version = "0.2", default-features = false }

[dev-dependencies]
serde_json = "1.0"

[features]
default = ["std", "heapsize", "serialize"]
default = ["std", "serialize"]
std = ["uint/std", "fixed-hash/std", "ethbloom/std", "primitive-types/std"]
heapsize = ["uint/heapsize", "primitive-types/heapsize", "fixed-hash/heapsize", "ethbloom/heapsize"]
serialize = ["std", "impl-serde", "primitive-types/serde", "ethbloom/serialize"]
5 changes: 2 additions & 3 deletions fixed-hash/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "fixed-hash"
version = "0.3.2"
version = "0.4.0"
authors = ["Parity Technologies <[email protected]>"]
license = "MIT"
homepage = "https://github.com/paritytech/parity-common"
Expand All @@ -10,10 +10,9 @@ documentation = "https://docs.rs/fixed-hash/"
readme = "README.md"

[package.metadata.docs.rs]
features = ["heapsize", "quickcheck", "api-dummy"]
features = ["quickcheck", "api-dummy"]

[dependencies]
heapsize = { version = "0.4", optional = true }
rand = { version = "0.5", optional = true, default-features = false }
rustc-hex = { version = "2.0", optional = true, default-features = false }
quickcheck = { version = "0.7", optional = true }
Expand Down
2 changes: 0 additions & 2 deletions fixed-hash/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,6 @@ fixed-hash = { version = "0.3", default-features = false }
- Enabled by default.
- `byteorder`: Provide API based on the `byteorder` crate.
- Enabled by default.
- `heapsize`: Provide `HeapsizeOf` implementation for hash types.
- Disabled by default.
- `quickcheck`: Provide `quickcheck` implementation for hash types.
- Disabled by default.
- `api-dummy`: Generate a dummy hash type for API documentation.
Expand Down
40 changes: 0 additions & 40 deletions fixed-hash/src/hash.rs
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,6 @@ macro_rules! construct_fixed_hash {
impl_rand_for_fixed_hash!($name);
impl_libc_for_fixed_hash!($name);
impl_rustc_hex_for_fixed_hash!($name);
impl_heapsize_for_fixed_hash!($name);
impl_quickcheck_for_fixed_hash!($name);
}
}
Expand Down Expand Up @@ -667,45 +666,6 @@ macro_rules! impl_rustc_hex_for_fixed_hash {
}
}

// Implementation for disabled heapsize crate support.
//
// # Note
//
// Feature guarded macro definitions instead of feature guarded impl blocks
// to work around the problems of introducing `heapsize` crate feature in
// a user crate.
#[cfg(not(
all(feature = "heapsize", not(target_os = "unknown"))
))]
#[macro_export]
#[doc(hidden)]
macro_rules! impl_heapsize_for_fixed_hash {
( $name:ident ) => {}
}

// Implementation for enabled heapsize crate support.
//
// # Note
//
// Feature guarded macro definitions instead of feature guarded impl blocks
// to work around the problems of introducing `heapsize` crate feature in
// a user crate.
#[cfg(
all(feature = "heapsize", not(target_os = "unknown"))
)]
#[macro_export]
#[doc(hidden)]
macro_rules! impl_heapsize_for_fixed_hash {
( $name:ident ) => {
impl $crate::heapsize::HeapSizeOf for $name {
#[inline]
fn heap_size_of_children(&self) -> usize {
0
}
}
}
}

// Implementation for disabled quickcheck crate support.
//
// # Note
Expand Down
4 changes: 0 additions & 4 deletions fixed-hash/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,6 @@ pub extern crate byteorder;
#[doc(hidden)]
pub mod libc {}

#[cfg(feature = "heapsize")]
#[doc(hidden)]
pub extern crate heapsize;

#[cfg(feature = "rustc-hex")]
#[doc(hidden)]
pub extern crate rustc_hex;
Expand Down
11 changes: 0 additions & 11 deletions fixed-hash/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -310,17 +310,6 @@ mod from_str {
}
}

#[cfg(all(
feature = "heapsize",
not(target_os = "unknown")
))]
#[test]
fn test_heapsizeof() {
use heapsize::HeapSizeOf;
let h = H128::zero();
assert_eq!(h.heap_size_of_children(), 0);
}

#[test]
fn from_h160_to_h256() {
let h160 = H160::from([
Expand Down
4 changes: 2 additions & 2 deletions keccak-hash/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "keccak-hash"
version = "0.2.0"
version = "0.3.0"
description = "`keccak-hash` is a set of utility functions to facilitate working with Keccak hashes (256/512 bits long)."
authors = ["Parity Technologies <[email protected]>"]
repository = "https://github.com/paritytech/parity-common"
Expand All @@ -9,7 +9,7 @@ license = "GPL-3.0"

[dependencies]
tiny-keccak = "1.4"
primitive-types = { path = "../primitive-types", version = "0.3" }
primitive-types = { path = "../primitive-types", version = "0.4" }

[dev-dependencies]
tempdir = "0.3"
2 changes: 1 addition & 1 deletion kvdb-rocksdb/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@ parity-rocksdb = "0.5"

[dev-dependencies]
tempdir = "0.3"
ethereum-types = { version = "0.6", path = "../ethereum-types" }
ethereum-types = { version = "0.7", path = "../ethereum-types" }
8 changes: 3 additions & 5 deletions parity-util-mem/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ malloc_size_of_derive = "0.1.0"
dlmalloc = { version = "0.1", features = ["global"], optional = true }
wee_alloc = { version = "0.4", optional = true }

elastic-array = { version = "*", optional = true }
ethereum-types = { version = "*", optional = true }
parking_lot = { version = "*", optional = true }
elastic-array = { version = "0", optional = true }
ethereum-types = { version = "0", optional = true }
parking_lot = { version = "0", optional = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should work fine, says cargo publish --dry-run


[target.'cfg(target_os = "windows")'.dependencies.winapi]
version = "0.3.4"
Expand All @@ -27,8 +27,6 @@ optional = true
[features]
default = ["std", "ethereum-impls"]
std = []
# when activated mem is removed through volatile primitive instead of clear_on_drop crate
volatile-erase = []
# use dlmalloc as global allocator
dlmalloc-global = ["dlmalloc", "estimate-heapsize"]
# use wee_alloc as global allocator
Expand Down
5 changes: 3 additions & 2 deletions parity-util-mem/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,14 @@ Collection of memory related utilities.

## Features
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a recommended way of using this crate to, for example, replace code that uses heapsizeof today and wishes to stop? :)


- volatile-erase : Not set by default, `Memzero` struct will be erasing memory through a simple [`write_volatile`](https://doc.rust-lang.org/std/ptr/fn.write_volatile.html) call.
- estimate-heapsize : Do not use allocator, but `size_of` or `size_of_val`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(commenting here because GH doesn't allow me to comment below)

Need to update the ## Dependency section too now that CoD is removed.


Others features define global allocator, see `src/alloc.rs`.

## Dependency

This crate groups common dependency, [`clear_on_drop`](https://crates.io/crates/clear_on_drop) is reexported, and a patched copy of unpublished [`malloc_size_of`](https://github.com/servo/servo/tree/master/components/malloc_size_of) from servo project is copied and partially reexported.
This crate groups common dependency, a patched copy of unpublished [`malloc_size_of`](https://github.com/servo/servo/tree/master/components/malloc_size_of) from servo project is copied and partially reexported.

`Malloc_size_of` code is used internally as a module with a few modification to be able to implement type locally.

For existing code using deprecated `HeapsizeOf` crate, calls to `heapsize_of_children` should be replace by calls to `size_of`.
2 changes: 0 additions & 2 deletions parity-util-mem/src/allocators.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,6 @@ use malloc_size::MallocUnconditionalSizeOf;
use std::os::raw::c_void;
#[cfg(not(feature = "std"))]
use core::ffi::c_void;
#[cfg(not(feature = "std"))]
use alloc::collections::btree_set::BTreeSet;

mod usable_size {

Expand Down
22 changes: 1 addition & 21 deletions parity-util-mem/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,12 @@ extern crate cfg_if;
#[cfg(not(feature = "std"))]
extern crate alloc;

extern crate clear_on_drop as cod;

#[macro_use] extern crate malloc_size_of_derive as malloc_size_derive;
extern crate malloc_size_of_derive as malloc_size_derive;

use std::ops::{Deref, DerefMut};

#[cfg(feature = "volatile-erase")]
use std::ptr;

#[cfg(not(feature = "volatile-erase"))]
pub use cod::clear::Clear;


cfg_if! {
if #[cfg(all(
feature = "jemalloc-global",
Expand Down Expand Up @@ -91,11 +84,6 @@ use core as std;
#[cfg(feature = "ethereum-impls")]
pub mod impls;

/// Reexport clear_on_drop crate.
pub mod clear_on_drop {
pub use cod::*;
}

pub use malloc_size_derive::*;
pub use malloc_size::{
MallocSizeOfOps,
Expand All @@ -115,7 +103,6 @@ impl<T: AsMut<[u8]>> From<T> for Memzero<T> {
}
}

#[cfg(feature = "volatile-erase")]
impl<T: AsMut<[u8]>> Drop for Memzero<T> {
fn drop(&mut self) {
unsafe {
Expand All @@ -126,13 +113,6 @@ impl<T: AsMut<[u8]>> Drop for Memzero<T> {
}
}

#[cfg(not(feature = "volatile-erase"))]
impl<T: AsMut<[u8]>> Drop for Memzero<T> {
fn drop(&mut self) {
self.as_mut().clear();
}
}

impl<T: AsMut<[u8]>> Deref for Memzero<T> {
type Target = T;

Expand Down
11 changes: 7 additions & 4 deletions parity-util-mem/src/malloc_size.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,6 @@
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;
#[cfg(not(feature = "std"))]
use alloc::string::String;
#[cfg(not(feature = "std"))]
mod std {
pub use core::*;
pub use alloc::collections;
Expand All @@ -60,7 +58,9 @@ mod std {
#[cfg(feature = "std")]
use std::sync::Arc;

use std::hash::{BuildHasher, Hash};
#[cfg(feature = "std")]
use std::hash::BuildHasher;
use std::hash::Hash;
use std::mem::size_of;
use std::ops::Range;
use std::ops::{Deref, DerefMut};
Expand All @@ -75,7 +75,7 @@ pub use alloc::boxed::Box;
pub type VoidPtrToSizeFn = unsafe extern "C" fn(ptr: *const c_void) -> usize;

/// A closure implementing a stateful predicate on pointers.
pub type VoidPtrToBoolFnMut = FnMut(*const c_void) -> bool;
pub type VoidPtrToBoolFnMut = dyn FnMut(*const c_void) -> bool;

/// Operations used when measuring heap usage of data structures.
pub struct MallocSizeOfOps {
Expand Down Expand Up @@ -211,6 +211,9 @@ pub mod inner_allocator_use {

use super::*;

#[cfg(not(feature = "std"))]
use alloc::string::String;

impl<T: ?Sized> MallocShallowSizeOf for Box<T> {
fn shallow_size_of(&self, ops: &mut MallocSizeOfOps) -> usize {
unsafe { ops.malloc_size_of(&**self) }
Expand Down
16 changes: 15 additions & 1 deletion parity-util-mem/src/sizeof.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,21 @@ use crate::malloc_size::{
MallocUnconditionalShallowSizeOf,
MallocSizeOfOps
};
#[cfg(not(feature = "std"))]
use alloc::boxed::Box;
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;
#[cfg(not(feature = "std"))]
use alloc::string::String;
#[cfg(not(feature = "std"))]
use core::mem::{size_of, size_of_val};
#[cfg(not(feature = "std"))]
use alloc::sync::Arc;

#[cfg(feature = "std")]
use std::mem::{size_of, size_of_val};
#[cfg(feature = "std")]
use std::sync::Arc;

impl<T: ?Sized> MallocShallowSizeOf for Box<T> {
fn shallow_size_of(&self, _ops: &mut MallocSizeOfOps) -> usize {
Expand All @@ -44,7 +58,7 @@ impl<T> MallocShallowSizeOf for Vec<T> {
}
}

impl<T> MallocUnconditionalShallowSizeOf for std::sync::Arc<T> {
impl<T> MallocUnconditionalShallowSizeOf for Arc<T> {
fn unconditional_shallow_size_of(&self, _ops: &mut MallocSizeOfOps) -> usize {
size_of::<T>()
}
Expand Down
7 changes: 3 additions & 4 deletions primitive-types/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,22 +1,21 @@
[package]
name = "primitive-types"
version = "0.3.0"
version = "0.4.0"
authors = ["Parity Technologies <[email protected]>"]
license = "Apache-2.0/MIT"
homepage = "https://github.com/paritytech/parity-common"
description = "Primitive types shared by Ethereum and Substrate"

[dependencies]
fixed-hash = { version = "0.3", path = "../fixed-hash", default-features = false }
uint = { version = "0.7", path = "../uint", default-features = false }
fixed-hash = { version = "0.4", path = "../fixed-hash", default-features = false }
uint = { version = "0.8", path = "../uint", default-features = false }
impl-serde = { version = "0.2", path = "impls/serde", default-features = false, optional = true }
impl-codec = { version = "0.2", path = "impls/codec", default-features = false, optional = true }
impl-rlp = { version = "0.2", path = "impls/rlp", default-features = false, optional = true }

[features]
default = ["std"]
std = ["uint/std", "fixed-hash/std", "impl-codec/std"]
heapsize = ["uint/heapsize", "fixed-hash/heapsize"]
byteorder = ["fixed-hash/byteorder"]
libc = ["fixed-hash/libc"]
rustc-hex = ["fixed-hash/rustc-hex"]
Expand Down
Loading