Skip to content

Implement ChunkStore and integrate it everywhere #6570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 4, 2024
Merged

Implement ChunkStore and integrate it everywhere #6570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
fb2d198
implement and integrate chunk store
teh-cmc Jun 4, 2024
0f4658b
support for overlapping chunks in latest-at queries
teh-cmc Jun 24, 2024
b8a41a9
link to text-log view issue
teh-cmc Jun 24, 2024
413fd92
sorted by time > sorted by row id
teh-cmc Jun 24, 2024
d7e90a7
missing emptiness checks
teh-cmc Jun 24, 2024
09daea0
deal with false is_sorted negatives
teh-cmc Jun 24, 2024
e4977d1
address review comments
teh-cmc Jun 24, 2024
f4d295f
lints
teh-cmc Jun 24, 2024
a861ff4
even more lints
teh-cmc Jun 24, 2024
d304e97
support per-component row-id ranges and fix static writes accordingly
teh-cmc Jul 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility.

## Technologies we use
### Apache Arrow
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_data_store`](crates/re_data_store/README.md).
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_chunk_store`](crates/re_chunk_store/README.md).

In Rust, we use the [`arrow2` crate](https://crates.io/crates/arrow2).

Expand Down Expand Up @@ -88,11 +88,11 @@ Of course, this will only take us so far. In the future we plan on caching queri
Here is an overview of the crates included in the project:

<picture>
<img src="https://static.rerun.io/crates/710562417756943b47657d3260622b5effe392f7/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/710562417756943b47657d3260622b5effe392f7/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/710562417756943b47657d3260622b5effe392f7/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/710562417756943b47657d3260622b5effe392f7/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/710562417756943b47657d3260622b5effe392f7/1200w.png">
<img src="https://static.rerun.io/crates/ea88def8bf549cd0d3abbe85ae787cf1a4db03bc/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/ea88def8bf549cd0d3abbe85ae787cf1a4db03bc/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/ea88def8bf549cd0d3abbe85ae787cf1a4db03bc/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/ea88def8bf549cd0d3abbe85ae787cf1a4db03bc/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/ea88def8bf549cd0d3abbe85ae787cf1a4db03bc/1200w.png">
</picture>


Expand Down Expand Up @@ -160,7 +160,7 @@ Update instructions:
| Crate | Description |
|----------------------|--------------------------------------------------------------------------|
| re_entity_db | In-memory storage of Rerun entities |
| re_query | Querying data in the re_data_store |
| re_query | Querying data in the re_chunk_store |
| re_types | The built-in Rerun data types, component types, and archetypes. |
| re_types_blueprint | The core traits and types that power Rerun's Blueprint sub-system. |
| re_log_encoding | Helpers for encoding and transporting Rerun log messages |
Expand All @@ -171,7 +171,7 @@ Update instructions:
| Crate | Description |
|-----------------|-----------------------------------------------------------------------------------------------|
| re_chunk | A chunk of Rerun data, encoded using Arrow. Used for logging, transport, storage and compute. |
| re_data_store | An in-memory time series database for Rerun log data, based on Apache Arrow. |
| re_chunk_store | An in-memory time series database for Rerun log data, based on Apache Arrow. |
| re_log_types | The basic building blocks of the Rerun data types and tables. |
| re_types_core | The core traits and types that power Rerun's data model. |
| re_format_arrow | Formatting of Apache Arrow tables. |
Expand Down
109 changes: 61 additions & 48 deletions Cargo.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4305,12 +4305,46 @@ dependencies = [
"re_tracing",
"re_tuid",
"re_types_core",
"serde",
"serde_bytes",
"similar-asserts",
"smallvec",
"static_assertions",
"thiserror",
]

[[package]]
name = "re_chunk_store"
version = "0.17.0-alpha.9"
dependencies = [
"ahash",
"anyhow",
"criterion",
"document-features",
"indent",
"insta",
"itertools 0.13.0",
"mimalloc",
"nohash-hasher",
"once_cell",
"parking_lot",
"rand",
"re_arrow2",
"re_chunk",
"re_format",
"re_format_arrow",
"re_log",
"re_log_types",
"re_tracing",
"re_types",
"re_types_core",
"similar-asserts",
"smallvec",
"thiserror",
"tinyvec",
"web-time",
]

[[package]]
name = "re_context_menu"
version = "0.17.0-alpha.9"
Expand Down Expand Up @@ -4356,6 +4390,7 @@ dependencies = [
"rayon",
"re_build_info",
"re_build_tools",
"re_chunk",
"re_log",
"re_log_encoding",
"re_log_types",
Expand Down Expand Up @@ -4383,37 +4418,6 @@ dependencies = [
"re_ws_comms",
]

[[package]]
name = "re_data_store"
version = "0.17.0-alpha.9"
dependencies = [
"ahash",
"anyhow",
"criterion",
"document-features",
"indent",
"insta",
"itertools 0.13.0",
"mimalloc",
"nohash-hasher",
"once_cell",
"parking_lot",
"rand",
"re_arrow2",
"re_format",
"re_format_arrow",
"re_log",
"re_log_types",
"re_tracing",
"re_types",
"re_types_core",
"similar-asserts",
"smallvec",
"thiserror",
"tinyvec",
"web-time",
]

[[package]]
name = "re_data_ui"
version = "0.17.0-alpha.9"
Expand All @@ -4426,7 +4430,7 @@ dependencies = [
"egui_plot",
"image",
"itertools 0.13.0",
"re_data_store",
"re_chunk_store",
"re_entity_db",
"re_error",
"re_format",
Expand Down Expand Up @@ -4498,7 +4502,8 @@ dependencies = [
"parking_lot",
"rand",
"re_build_info",
"re_data_store",
"re_chunk",
"re_chunk_store",
"re_format",
"re_int_histogram",
"re_log",
Expand Down Expand Up @@ -4574,6 +4579,7 @@ dependencies = [
"mimalloc",
"parking_lot",
"re_build_info",
"re_chunk",
"re_log",
"re_log_types",
"re_smart_channel",
Expand Down Expand Up @@ -4666,7 +4672,8 @@ dependencies = [
"paste",
"rand",
"re_arrow2",
"re_data_store",
"re_chunk",
"re_chunk_store",
"re_error",
"re_format",
"re_log",
Expand Down Expand Up @@ -4770,8 +4777,8 @@ dependencies = [
"re_build_info",
"re_build_tools",
"re_chunk",
"re_chunk_store",
"re_data_loader",
"re_data_store",
"re_log",
"re_log_encoding",
"re_log_types",
Expand Down Expand Up @@ -4811,8 +4818,9 @@ dependencies = [
"itertools 0.13.0",
"nohash-hasher",
"once_cell",
"re_chunk",
"re_chunk_store",
"re_context_menu",
"re_data_store",
"re_data_ui",
"re_entity_db",
"re_log",
Expand Down Expand Up @@ -4850,7 +4858,7 @@ dependencies = [
"ahash",
"egui",
"nohash-hasher",
"re_data_store",
"re_chunk_store",
"re_entity_db",
"re_log",
"re_log_types",
Expand All @@ -4868,7 +4876,7 @@ version = "0.17.0-alpha.9"
dependencies = [
"egui",
"egui_plot",
"re_data_store",
"re_chunk_store",
"re_entity_db",
"re_log",
"re_log_types",
Expand All @@ -4887,7 +4895,7 @@ version = "0.17.0-alpha.9"
dependencies = [
"egui",
"egui_extras",
"re_data_store",
"re_chunk_store",
"re_data_ui",
"re_entity_db",
"re_log_types",
Expand All @@ -4914,7 +4922,7 @@ dependencies = [
"mimalloc",
"nohash-hasher",
"once_cell",
"re_data_store",
"re_chunk_store",
"re_data_ui",
"re_entity_db",
"re_error",
Expand Down Expand Up @@ -4944,7 +4952,7 @@ dependencies = [
"egui",
"half 2.3.1",
"ndarray",
"re_data_store",
"re_chunk_store",
"re_data_ui",
"re_entity_db",
"re_log",
Expand All @@ -4967,7 +4975,7 @@ version = "0.17.0-alpha.9"
dependencies = [
"egui",
"egui_commonmark",
"re_data_store",
"re_chunk_store",
"re_renderer",
"re_space_view",
"re_tracing",
Expand All @@ -4982,7 +4990,7 @@ version = "0.17.0-alpha.9"
dependencies = [
"egui",
"egui_extras",
"re_data_store",
"re_chunk_store",
"re_data_ui",
"re_entity_db",
"re_log",
Expand All @@ -5004,7 +5012,7 @@ dependencies = [
"egui_plot",
"itertools 0.13.0",
"rayon",
"re_data_store",
"re_chunk_store",
"re_format",
"re_log",
"re_log_types",
Expand Down Expand Up @@ -5036,8 +5044,8 @@ version = "0.17.0-alpha.9"
dependencies = [
"egui",
"itertools 0.13.0",
"re_chunk_store",
"re_context_menu",
"re_data_store",
"re_data_ui",
"re_entity_db",
"re_format",
Expand Down Expand Up @@ -5220,9 +5228,10 @@ dependencies = [
"re_blueprint_tree",
"re_build_info",
"re_build_tools",
"re_chunk",
"re_chunk_store",
"re_data_loader",
"re_data_source",
"re_data_store",
"re_data_ui",
"re_edit_ui",
"re_entity_db",
Expand Down Expand Up @@ -5294,8 +5303,9 @@ dependencies = [
"nohash-hasher",
"once_cell",
"parking_lot",
"re_chunk",
"re_chunk_store",
"re_data_source",
"re_data_store",
"re_entity_db",
"re_error",
"re_format",
Expand Down Expand Up @@ -5354,7 +5364,8 @@ dependencies = [
"nohash-hasher",
"once_cell",
"parking_lot",
"re_data_store",
"re_chunk",
"re_chunk_store",
"re_entity_db",
"re_log",
"re_log_types",
Expand Down Expand Up @@ -5482,6 +5493,7 @@ dependencies = [
"re_analytics",
"re_build_info",
"re_build_tools",
"re_chunk",
"re_crash_handler",
"re_data_source",
"re_entity_db",
Expand All @@ -5498,6 +5510,7 @@ dependencies = [
"re_viewer",
"re_web_viewer_server",
"re_ws_comms",
"similar-asserts",
]

[[package]]
Expand Down
1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ re_build_info = { path = "crates/re_build_info", version = "=0.17.0-alpha.9", de
re_build_tools = { path = "crates/re_build_tools", version = "=0.17.0-alpha.9", default-features = false }
re_case = { path = "crates/re_case", version = "=0.17.0-alpha.9", default-features = false }
re_chunk = { path = "crates/re_chunk", version = "=0.17.0-alpha.9", default-features = false }
re_chunk_store = { path = "crates/re_chunk_store", version = "=0.17.0-alpha.9", default-features = false }
re_context_menu = { path = "crates/re_context_menu", version = "=0.17.0-alpha.9", default-features = false }
re_crash_handler = { path = "crates/re_crash_handler", version = "=0.17.0-alpha.9", default-features = false }
re_data_loader = { path = "crates/re_data_loader", version = "=0.17.0-alpha.9", default-features = false }
Expand Down
2 changes: 1 addition & 1 deletion crates/re_analytics/src/event.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ pub struct OpenRecording {
pub data_source: Option<&'static str>,
}

/// Basic information about a recording's data store.
/// Basic information about a recording's chunk store.
pub struct StoreInfo {
/// Name of the application.
///
Expand Down
19 changes: 16 additions & 3 deletions crates/re_chunk/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,16 @@ all-features = true
[features]
default = []

## Enable (de)serialization using serde.
serde = [
"dep:serde",
"dep:serde_bytes",
"re_log_types/serde",
"re_string_interner/serde",
"re_tuid/serde",
"re_types_core/serde",
]


[dependencies]

Expand All @@ -40,10 +50,8 @@ re_types_core.workspace = true
ahash.workspace = true
anyhow.workspace = true
arrow2 = { workspace = true, features = [
"io_ipc",
"io_print",
"compute_comparison",
"compute_concatenate",
"compute_filter",
] }
backtrace.workspace = true
document-features.workspace = true
Expand All @@ -55,12 +63,17 @@ smallvec.workspace = true
static_assertions.workspace = true
thiserror.workspace = true

# Optional dependencies:
serde = { workspace = true, optional = true, features = ["derive", "rc"] }
serde_bytes = { workspace = true, optional = true }

# Native dependencies:
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
crossbeam.workspace = true


[dev-dependencies]
re_log = { workspace = true, features = ["setup"] }
criterion.workspace = true
mimalloc.workspace = true
similar-asserts.workspace = true
Loading
Loading