Skip to content

feat(pipeline): support Loki API #6390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jun 28, 2025

Conversation

shuiyisong
Copy link
Contributor

@shuiyisong shuiyisong commented Jun 25, 2025

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

close #4800

What's changed and what's your intention?

This PR adds support for the Loki API to run pipelines.
The process becomes a little complicated; see the diagram below for a quick glimpse.

Untitled Diagram

For entering the pipeline process

  1. The original ts (in nanosecond timestamp datatype), Loki text line, and all the labels and structure metadata are merged into one big map
  2. This map is used as the input data of the pipeline

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

@shuiyisong shuiyisong requested a review from a team as a code owner June 25, 2025 00:14
@github-actions github-actions bot added size/M docs-not-required This change does not impact docs. labels Jun 25, 2025
@sunng87 sunng87 linked an issue Jun 25, 2025 that may be closed by this pull request
@shuiyisong shuiyisong requested review from sunng87 and paomian June 25, 2025 03:28
@killme2008 killme2008 requested a review from Copilot June 25, 2025 12:30
Copilot

This comment was marked as outdated.

@killme2008 killme2008 requested a review from Copilot June 27, 2025 02:58
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds support for running pipelines on incoming Loki payloads by routing HTTP ingest through the pipeline engine when a pipeline header is present, introducing iterator-based JSON/Protobuf parsers, and validating functionality with new integration tests.

  • Route loki_ingest through run_pipeline when GREPTIME_PIPELINE_NAME_HEADER_NAME is provided.
  • Implement LokiJsonParser/LokiPbParser and extract_item to convert streams into generic item iterators.
  • Include new integration tests (test_loki_pb_logs_with_pipeline, test_loki_json_logs_with_pipeline) for the pipeline path.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests-integration/tests/http.rs Added two integration tests covering PB and JSON payloads with pipeline execution.
src/servers/src/http/loki.rs Refactored ingest handler to support optional pipeline, extracted parsing logic into dedicated parsers and iterator adapters.
src/pipeline/src/lib.rs Exported Timestamp in the public API and removed now-unused unwrap_or_warn_continue macro.
Comments suppressed due to low confidence (3)

src/pipeline/src/lib.rs:52

  • The unwrap_or_warn_continue macro has been removed; ensure no downstream code references it or update consumers accordingly.
}

tests-integration/tests/http.rs:3962

  • [nitpick] Consider adding negative or malformed-payload tests (e.g., invalid pipeline name or bad YAML) to cover failure paths for the pipeline ingest logic.
pub async fn test_loki_pb_logs_with_pipeline(store_type: StorageType) {

src/servers/src/http/loki.rs:55

  • [nitpick] Verify that use crate::pipeline::run_pipeline points to the intended module and consider unifying imports (e.g., consistently using either pipeline::run_pipeline or crate::pipeline::run_pipeline) for clarity.
use crate::pipeline::run_pipeline;

Signed-off-by: shuiyisong <[email protected]>
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@killme2008 killme2008 added this pull request to the merge queue Jun 28, 2025
Merged via the queue into GreptimeTeam:main with commit dec8c52 Jun 28, 2025
41 checks passed
@evenyag evenyag added the backport This patch is to be backported to current release branch label Jul 2, 2025
@evenyag evenyag mentioned this pull request Jul 2, 2025
18 tasks
evenyag pushed a commit to evenyag/greptimedb that referenced this pull request Jul 2, 2025
* chore: use schema_info

Signed-off-by: shuiyisong <[email protected]>

* refactor: abstract loki item generator

Signed-off-by: shuiyisong <[email protected]>

* chore: introduce middle item

Signed-off-by: shuiyisong <[email protected]>

* feat: introduce pipeline in loki api

Signed-off-by: shuiyisong <[email protected]>

* test: add tests

Signed-off-by: shuiyisong <[email protected]>

* chore: minor update

Signed-off-by: shuiyisong <[email protected]>

* chore: minor update

Signed-off-by: shuiyisong <[email protected]>

* chore: update prefix and test

Signed-off-by: shuiyisong <[email protected]>

* chore: change recursion to loop

Signed-off-by: shuiyisong <[email protected]>

* fix: cr issue

Signed-off-by: shuiyisong <[email protected]>

---------

Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: evenyag <[email protected]>
evenyag pushed a commit that referenced this pull request Jul 3, 2025
* chore: use schema_info

Signed-off-by: shuiyisong <[email protected]>

* refactor: abstract loki item generator

Signed-off-by: shuiyisong <[email protected]>

* chore: introduce middle item

Signed-off-by: shuiyisong <[email protected]>

* feat: introduce pipeline in loki api

Signed-off-by: shuiyisong <[email protected]>

* test: add tests

Signed-off-by: shuiyisong <[email protected]>

* chore: minor update

Signed-off-by: shuiyisong <[email protected]>

* chore: minor update

Signed-off-by: shuiyisong <[email protected]>

* chore: update prefix and test

Signed-off-by: shuiyisong <[email protected]>

* chore: change recursion to loop

Signed-off-by: shuiyisong <[email protected]>

* fix: cr issue

Signed-off-by: shuiyisong <[email protected]>

---------

Signed-off-by: shuiyisong <[email protected]>
Signed-off-by: evenyag <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport This patch is to be backported to current release branch docs-not-required This change does not impact docs. size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Loki ingestion with greptime pipeline
4 participants