feat(mito): bulk insert request handling on datanode#5831
Merged
v0y4g3r merged 8 commits intoGreptimeTeam:mainfrom Apr 15, 2025
Merged
feat(mito): bulk insert request handling on datanode#5831v0y4g3r merged 8 commits intoGreptimeTeam:mainfrom
v0y4g3r merged 8 commits intoGreptimeTeam:mainfrom
Conversation
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
v0y4g3r
force-pushed
the
feat/bulk-insert
branch
2 times, most recently
from
ee9738e to
0bdb1ce
Compare
evenyag
reviewed
Apr 9, 2025
evenyag
approved these changes
Apr 10, 2025
Contributor
evenyag
left a comment
evenyag
left a comment
There was a problem hiding this comment.
We can merge it and test the performance first.
11 tasks
Contributor
There was a problem hiding this comment.
Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.
Comments suppressed due to low confidence (1)
src/mito2/src/worker/handle_bulk_insert.rs:32
- Consider adding unit tests to verify the bulk insert handling logic in this function, covering both success and error paths.
pub(crate) async fn handle_bulk_inserts(
waynexia
reviewed
Apr 15, 2025
### Add Error Handling and Enhance Bulk Insert Functionality - **Error Handling**: Introduced a new error variant `ConvertDataType` in `error.rs` to handle conversion failures from `ConcreteDataType` to `ColumnDataType`. - **Bulk Insert Enhancements**: - Updated `WorkerRequest::BulkInserts` in `request.rs` to include metadata and sender. - Implemented `handle_bulk_inserts` in `worker.rs` to process bulk insert requests with region metadata. - Added functions `region_metadata_to_column_schema` and `record_batch_to_rows` in `handle_bulk_insert.rs` for schema conversion and row processing. - **API Changes**: Modified `RegionBulkInsertsRequest` in `region_request.rs` to include `region_id`. Files affected: `error.rs`, `request.rs`, `worker.rs`, `handle_bulk_insert.rs`, `region_request.rs`.
**Enhance Error Handling and Add Unit Tests** - Improved error handling in `record_batch_to_rows` function within `handle_bulk_insert.rs` by returning `Result` and handling errors with `context`. - Added unit tests for `region_metadata_to_column_schema` and `record_batch_to_rows` functions in `handle_bulk_insert.rs` to ensure correct functionality and error handling.
- **Refactor Error Handling**: Updated error handling in `error.rs` by modifying the `ConvertDataType` error handling. - **Improve Logging and Error Reporting**: Enhanced logging and error reporting in `worker.rs` by adding error messages for missing region metadata. - **Add New Error Type**: Introduced `DecodeArrowIpc` error in `metadata.rs` to handle Arrow IPC decoding failures. - **Handle Arrow IPC Decoding**: Updated `region_request.rs` to handle Arrow IPC decoding errors using the new `DecodeArrowIpc` error type.
Refactor `handle_bulk_insert.rs` to simplify row construction - Removed the mutable `current_row` vector and refactored `row_at` function to return a new vector directly. - Updated `record_batch_to_rows` to utilize the refactored `row_at` function for constructing rows.
### Commit Summary **Enhancements in Region Server Request Handling** - Updated `region_server.rs` to include `RegionRequest::BulkInserts(_)` in the `RegionChange::Ingest` category, improving the handling of bulk insert operations. - Refined the categorization of region requests to ensure accurate mapping to `RegionChange` actions.
v0y4g3r
force-pushed
the
feat/bulk-insert
branch
from
c797e58 to
567a541
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
Naive implementation of bulk insert on datanode that simply bridges bulk inserts to row-wise insert API.
PR Checklist
Please convert it to a draft if some of the following conditions are not met.