-
Notifications
You must be signed in to change notification settings - Fork 384
feat(pipeline): support Loki API #6390
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(pipeline): support Loki API #6390
Conversation
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: shuiyisong <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Adds support for running pipelines on incoming Loki payloads by routing HTTP ingest through the pipeline engine when a pipeline header is present, introducing iterator-based JSON/Protobuf parsers, and validating functionality with new integration tests.
- Route
loki_ingest
throughrun_pipeline
whenGREPTIME_PIPELINE_NAME_HEADER_NAME
is provided. - Implement
LokiJsonParser
/LokiPbParser
andextract_item
to convert streams into generic item iterators. - Include new integration tests (
test_loki_pb_logs_with_pipeline
,test_loki_json_logs_with_pipeline
) for the pipeline path.
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
File | Description |
---|---|
tests-integration/tests/http.rs | Added two integration tests covering PB and JSON payloads with pipeline execution. |
src/servers/src/http/loki.rs | Refactored ingest handler to support optional pipeline, extracted parsing logic into dedicated parsers and iterator adapters. |
src/pipeline/src/lib.rs | Exported Timestamp in the public API and removed now-unused unwrap_or_warn_continue macro. |
Comments suppressed due to low confidence (3)
src/pipeline/src/lib.rs:52
- The
unwrap_or_warn_continue
macro has been removed; ensure no downstream code references it or update consumers accordingly.
}
tests-integration/tests/http.rs:3962
- [nitpick] Consider adding negative or malformed-payload tests (e.g., invalid pipeline name or bad YAML) to cover failure paths for the pipeline ingest logic.
pub async fn test_loki_pb_logs_with_pipeline(store_type: StorageType) {
src/servers/src/http/loki.rs:55
- [nitpick] Verify that
use crate::pipeline::run_pipeline
points to the intended module and consider unifying imports (e.g., consistently using eitherpipeline::run_pipeline
orcrate::pipeline::run_pipeline
) for clarity.
use crate::pipeline::run_pipeline;
Signed-off-by: shuiyisong <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
* chore: use schema_info Signed-off-by: shuiyisong <[email protected]> * refactor: abstract loki item generator Signed-off-by: shuiyisong <[email protected]> * chore: introduce middle item Signed-off-by: shuiyisong <[email protected]> * feat: introduce pipeline in loki api Signed-off-by: shuiyisong <[email protected]> * test: add tests Signed-off-by: shuiyisong <[email protected]> * chore: minor update Signed-off-by: shuiyisong <[email protected]> * chore: minor update Signed-off-by: shuiyisong <[email protected]> * chore: update prefix and test Signed-off-by: shuiyisong <[email protected]> * chore: change recursion to loop Signed-off-by: shuiyisong <[email protected]> * fix: cr issue Signed-off-by: shuiyisong <[email protected]> --------- Signed-off-by: shuiyisong <[email protected]> Signed-off-by: evenyag <[email protected]>
* chore: use schema_info Signed-off-by: shuiyisong <[email protected]> * refactor: abstract loki item generator Signed-off-by: shuiyisong <[email protected]> * chore: introduce middle item Signed-off-by: shuiyisong <[email protected]> * feat: introduce pipeline in loki api Signed-off-by: shuiyisong <[email protected]> * test: add tests Signed-off-by: shuiyisong <[email protected]> * chore: minor update Signed-off-by: shuiyisong <[email protected]> * chore: minor update Signed-off-by: shuiyisong <[email protected]> * chore: update prefix and test Signed-off-by: shuiyisong <[email protected]> * chore: change recursion to loop Signed-off-by: shuiyisong <[email protected]> * fix: cr issue Signed-off-by: shuiyisong <[email protected]> --------- Signed-off-by: shuiyisong <[email protected]> Signed-off-by: evenyag <[email protected]>
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
close #4800
What's changed and what's your intention?
This PR adds support for the Loki API to run pipelines.
The process becomes a little complicated; see the diagram below for a quick glimpse.
For entering the pipeline process
PR Checklist
Please convert it to a draft if some of the following conditions are not met.