Skip to content

feat: add time series statistics profile #17809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

feat: add time series statistics profile #17809

wants to merge 13 commits into from
+729 −4

Conversation

dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Apr 18, 2025 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

close: #17779

Add time series statistics profile. Iniaitily, we support output_rows and output_bytes. This allows us to visualize and compare the data processing speed of different plan nodes.

Disabled in log by default. Enable by adding databend::log::time_series=INFO

[log.file]
level = "INFO,databend::log::time_series=INFO"

{
  "query_id": "76f9fa48-7843-456b-9519-f53daae9a1d7",
  "plans": [
    {
      "plan_id": 0,
      "data": [
        [[
            1745932982,5
        ]],
        [[
            1745932982,40
        ]]
      ]
    },
    {
      "plan_id": 1,
      "data": [
        [[
            1745932982,5
        ]],
        [[
            1745932982,40
        ]]
      ]
    },
    {
      "plan_id": 2,
      "data": [
        [[
            1745932980,20512768,55312384,24174848
        ]],
        [[
            1745932980,164102144,442499072,193398784
        ]]
      ]
    }
  ],
  "desc": [
    {
      "name": "OutputRows",
      "index": 0
    },
    {
      "name": "OutputBytes",
      "index": 1
    }
  ]
}

Output Format

We use a compressed representation for time series data points to save space and improve transmission efficiency. Instead of storing each (timestamp, value) pair separately, consecutive data points with incrementing timestamps are grouped into segments. Each segment is represented as a list where the first element is the starting timestamp, and the subsequent elements are the values for each consecutive timestamp in that segment.

For example, the raw data
[(t0, v0), (t1, v1), (t2, v2), (t4, v4), (t5, v5)]
will be compressed as
[[t0, v0, v1, v2], [t4, v4, v5]].

The timestamp is the unix timestamp in second

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Apr 18, 2025
@dqhl76 dqhl76 force-pushed the time-series-profile branch 2 times, most recently from 971528e to dfdef76 Compare April 18, 2025 13:17
@dqhl76 dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 21, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17809-bf4ac89-1745211953

note: this image tag is only available for internal use.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@dqhl76
Copy link
Collaborator Author

dqhl76 commented Apr 21, 2025

Disabled in log by default. Enable by adding databend::log::time_series=INFO

[log.file]
level = "INFO,databend::log::time_series=INFO"

@dqhl76 dqhl76 force-pushed the time-series-profile branch from 3cf1865 to a0ecd35 Compare April 21, 2025 06:38
@dqhl76 dqhl76 marked this pull request as ready for review April 21, 2025 07:36
@dqhl76 dqhl76 requested a review from zhang2014 April 21, 2025 07:36
@dqhl76 dqhl76 added the ci-benchmark Benchmark: run all test label Apr 22, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17809-07566f6-1745306957

note: this image tag is only available for internal use.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@dqhl76 dqhl76 removed the ci-benchmark Benchmark: run all test label Apr 22, 2025
@dqhl76
Copy link
Collaborator Author

dqhl76 commented Apr 22, 2025

@zhang2014 Benchmark looks good. Please take a look again

@zhang2014 zhang2014 marked this pull request as draft April 24, 2025 02:52
@dqhl76 dqhl76 force-pushed the time-series-profile branch from 5afa43f to 2ae2c1c Compare April 28, 2025 06:10
@dqhl76 dqhl76 added the ci-benchmark Benchmark: run all test label Apr 28, 2025
@dqhl76 dqhl76 removed the ci-benchmark Benchmark: run all test label Apr 28, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17809-6e97794-1745831680

note: this image tag is only available for internal use.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@dqhl76 dqhl76 marked this pull request as ready for review April 30, 2025 01:42
@dqhl76 dqhl76 requested a review from zhang2014 April 30, 2025 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhang2014 zhang2014 zhang2014 left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature: support time-series plan profile
2 participants
@dqhl76 @zhang2014