perf: Optimize Grafana query for trip view to leverage indexes more effectively #4964

jaypark0006 · 2025-09-24T04:55:58Z

This PR replaces the original positions aggregation query that used OR + subquery with a UNION ALL based query shape. The new form preserves the same result but allows PostgreSQL to leverage indexes more effectively.

old version:

SELECT
	$__timeGroup(date, '5s') AS time,
	avg(latitude) AS latitude,
	avg(longitude) AS longitude
FROM
	positions
WHERE
  car_id = $car_id and (drive_id in (select id from drives where $__timeFilter(start_date)) or drive_id is null and $__timeFilter(date))
GROUP BY
	1
ORDER BY
	1 ASC

new version:

SELECT
  t.time AS time,
  avg(t.latitude)  AS latitude,
  avg(t.longitude) AS longitude
FROM (
  SELECT
    $__timeGroup(p.date, '5s') AS time,
    avg(p.latitude)  AS latitude,
    avg(p.longitude) AS longitude
  FROM positions p
  JOIN drives d ON d.id = p.drive_id
  WHERE p.car_id = $car_id
    AND $__timeFilter(d.start_date)
  GROUP BY 1

  UNION ALL

  SELECT
    $__timeGroup(p.date, '5s') AS time,
    avg(p.latitude)  AS latitude,
    avg(p.longitude) AS longitude
  FROM positions p
  WHERE p.car_id = $car_id
    AND p.drive_id IS NULL
    AND $__timeFilter(p.date)
  GROUP BY 1
) AS t
GROUP BY 1
ORDER BY 1 ASC

for instance:
old version:

SELECT floor(extract(epoch from date) / 5) * 5 AS time,
       avg(latitude)                           AS latitude,
       avg(longitude)                          AS longitude
FROM positions
WHERE car_id = '1'
  and (drive_id in (select id from drives where start_date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z')
    or drive_id is null and date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z')
GROUP BY 1
ORDER BY 1 ASC

new version:

SELECT
  t.time AS time,
  avg(t.latitude)  AS latitude,
  avg(t.longitude) AS longitude
FROM (
  SELECT
    floor(extract(epoch from date) / 5) * 5 AS time,
    avg(p.latitude)  AS latitude,
    avg(p.longitude) AS longitude
  FROM positions p
  JOIN drives d ON d.id = p.drive_id
  WHERE p.car_id = '1'
    AND start_date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
  GROUP BY 1

  UNION ALL

  SELECT
    floor(extract(epoch from date) / 5) * 5 AS time,
    avg(p.latitude)  AS latitude,
    avg(p.longitude) AS longitude
  FROM positions p
  WHERE p.car_id = '1'
    AND p.drive_id IS NULL
    AND date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
  GROUP BY 1
) AS t
GROUP BY 1
ORDER BY 1 ASC

explain:
old version:

Finalize GroupAggregate  (cost=445244.60..1011242.59 rows=3871731 width=96)
  Group Key: ((floor((EXTRACT(epoch FROM positions.date) / '5'::numeric)) * '5'::numeric))
  ->  Gather Merge  (cost=445244.60..874118.79 rows=3226442 width=96)
        Workers Planned: 2
        ->  Partial GroupAggregate  (cost=444244.57..500707.31 rows=1613221 width=96)
              Group Key: ((floor((EXTRACT(epoch FROM positions.date) / '5'::numeric)) * '5'::numeric))
              ->  Sort  (cost=444244.57..448277.63 rows=1613221 width=48)
                    Sort Key: ((floor((EXTRACT(epoch FROM positions.date) / '5'::numeric)) * '5'::numeric))
                    ->  Parallel Seq Scan on positions  (cost=130.23..178656.29 rows=1613221 width=48)
                          Filter: ((car_id = '1'::smallint) AND ((ANY (drive_id = (hashed SubPlan 1).col1)) OR ((drive_id IS NULL) AND (date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (date <= '2025-09-23 16:00:00'::timestamp without time zone))))
                          SubPlan 1
                            ->  Seq Scan on drives  (cost=0.00..130.22 rows=6 width=4)
                                  Filter: ((start_date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (start_date <= '2025-09-23 16:00:00'::timestamp without time zone))
JIT:
  Functions: 23
"  Options: Inlining true, Optimization true, Expressions true, Deforming true"

new version:

Sort  (cost=39751.44..39751.94 rows=200 width=96)
  Sort Key: ((floor((EXTRACT(epoch FROM p.date) / '5'::numeric)) * '5'::numeric))
  ->  HashAggregate  (cost=39740.79..39743.79 rows=200 width=96)
        Group Key: ((floor((EXTRACT(epoch FROM p.date) / '5'::numeric)) * '5'::numeric))
        ->  Append  (cost=38826.46..39603.12 rows=18356 width=96)
              ->  HashAggregate  (cost=38826.46..39282.31 rows=18234 width=96)
                    Group Key: (floor((EXTRACT(epoch FROM p.date) / '5'::numeric)) * '5'::numeric)
                    ->  Nested Loop  (cost=0.43..38689.71 rows=18234 width=48)
                          ->  Seq Scan on drives d  (cost=0.00..130.22 rows=6 width=4)
                                Filter: ((start_date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (start_date <= '2025-09-23 16:00:00'::timestamp without time zone))
                          ->  Index Scan using positions_drive_id_date_index on positions p  (cost=0.43..6357.27 rows=3892 width=28)
                                Index Cond: (drive_id = d.id)
                                Filter: (car_id = '1'::smallint)
              ->  HashAggregate  (cost=225.98..229.03 rows=122 width=96)
                    Group Key: (floor((EXTRACT(epoch FROM p_1.date) / '5'::numeric)) * '5'::numeric)
                    ->  Index Scan using positions_drive_id_date_index on positions p_1  (cost=0.43..225.07 rows=122 width=48)
                          Index Cond: ((drive_id IS NULL) AND (date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (date <= '2025-09-23 16:00:00'::timestamp without time zone))
                          Filter: (car_id = '1'::smallint)

netlify · 2025-09-24T04:56:54Z

✅ Deploy Preview for teslamate ready!

Name	Link
🔨 Latest commit	`081cecd`
🔍 Latest deploy log	https://app.netlify.com/projects/teslamate/deploys/68dea42c825bc6000892a65d
😎 Deploy Preview	https://deploy-preview-4964--teslamate.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

JakobLichterfeld · 2025-09-25T08:16:49Z

Nice find, and thanks for your contribution!
Maybe Swiffer can take a look.

swiffer · 2025-09-27T11:10:49Z

🚀 nice finding!

this one looks even better for me while may is easier to read / understand ? @jaypark0006 - could you retest with that one?

SELECT floor(extract(epoch from date) / 5) * 5 AS time,
       avg(latitude)                           AS latitude,
       avg(longitude)                          AS longitude
from positions
where
  car_id = '2'
  and date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
  and (drive_id is null or drive_id in (select id from drives where start_date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'))
GROUP BY 1
ORDER BY 1 ASC

jaypark0006 · 2025-10-01T00:54:58Z

🚀 nice finding!

this one looks even better for me while may is easier to read / understand ? @jaypark0006 - could you retest with that one?

SELECT floor(extract(epoch from date) / 5) * 5 AS time,
       avg(latitude)                           AS latitude,
       avg(longitude)                          AS longitude
from positions
where
  car_id = '2'
  and date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
  and (drive_id is null or drive_id in (select id from drives where start_date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'))
GROUP BY 1
ORDER BY 1 ASC

Hi, sorry for the long wait, I finally got the rest today.

Your version’s condition is different from the original one.

In the original, date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z' is not a global/common condition (see #4791). I believe Matthias Wirtz wanted to capture the full drive positions even if the drive itself extends beyond this time range.

jaypark0006 · 2025-10-01T01:09:12Z

adding more detail on why the conditions differ and what we can do to keep semantics while improving index usage.

Before

WHERE car_id = '1'
  AND date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'

#4791

WHERE car_id = '1'
  AND (
        drive_id IN (
          SELECT id FROM drives
          WHERE start_date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
        )
      OR (
          drive_id IS NULL
          AND date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
      )
  )

PR version

-- branch A
WHERE p.car_id = '1'
  AND start_date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
GROUP BY 1

UNION ALL

-- branch B
WHERE p.car_id = '1'
  AND p.drive_id IS NULL
  AND date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
GROUP BY 1

swiffer · 2025-10-01T05:11:21Z

🚀 nice finding!
this one looks even better for me while may is easier to read / understand ? @jaypark0006 - could you retest with that one?
SELECT floor(extract(epoch from date) / 5) * 5 AS time,
       avg(latitude)                           AS latitude,
       avg(longitude)                          AS longitude
from positions
where
  car_id = '2'
  and date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'
  and (drive_id is null or drive_id in (select id from drives where start_date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z'))
GROUP BY 1
ORDER BY 1 ASC
Hi, sorry for the long wait, I finally got the rest today.

Your version’s condition is different from the original one.

In the original, date BETWEEN '2025-09-22T16:00:00Z' AND '2025-09-23T16:00:00Z' is not a global/common condition (see #4791). I believe Matthias Wirtz wanted to capture the full drive positions even if the drive itself extends beyond this time range.

it's me ;) - yes, you're absolutely right. i wanted to avoid showing different data in different panels - as for the other drives we filter by start date only i wanted to show the full drives positions in the map as well.

with unioned_positions as (

	-- fetch all positions based on start_date of drives so the map aligns with data shown in other panels
	select p.* from positions p
	inner join drives d on p.drive_id = d.id
	where p.car_id = '2' and start_date between '2025-09-22T16:00:00Z' and '2025-09-23T16:00:00Z'
    
    union all
    
    -- get all positions logged while not driving
    select * from positions p
	where p.car_id = '2' and drive_id is null and date between '2025-09-22T16:00:00Z' and '2025-09-23T16:00:00Z'

)

SELECT floor(extract(epoch from date) / 5) * 5 AS time,
       avg(latitude)                           AS latitude,
       avg(longitude)                          AS longitude
from unioned_positions
GROUP BY 1
ORDER BY 1 ASC

this one should work just fine, right?

jaypark0006 · 2025-10-01T07:09:24Z

@swiffer Yes, your last version is correct. I think it works well in my database and also avoids the double grouping.

Signed-off-by: jaypark0006 <[email protected]>

jaypark0006 · 2025-10-02T16:18:53Z

Hi @swiffer I tested your new SQL with the same dataset. The results are consistent, and the performance is also the same. Could you please review it again?

double group

Sort  (cost=40470.08..40470.58 rows=200 width=96) (actual time=24.210..24.256 rows=1054 loops=1)
  Sort Key: ((floor((EXTRACT(epoch FROM p.date) / '5'::numeric)) * '5'::numeric))
  Sort Method: quicksort  Memory: 102kB
  ->  HashAggregate  (cost=40459.44..40462.44 rows=200 width=96) (actual time=23.123..23.862 rows=1054 loops=1)
        Group Key: ((floor((EXTRACT(epoch FROM p.date) / '5'::numeric)) * '5'::numeric))
        Batches: 1  Memory Usage: 849kB
        ->  Append  (cost=39673.67..40319.40 rows=18672 width=96) (actual time=20.918..22.058 rows=1056 loops=1)
              ->  HashAggregate  (cost=39673.67..40139.37 rows=18628 width=96) (actual time=20.917..21.750 rows=971 loops=1)
                    Group Key: (floor((EXTRACT(epoch FROM p.date) / '5'::numeric)) * '5'::numeric)
                    Batches: 1  Memory Usage: 1169kB
                    ->  Nested Loop  (cost=0.43..39533.96 rows=18628 width=48) (actual time=0.048..15.015 rows=14747 loops=1)
                          ->  Seq Scan on drives d  (cost=0.00..130.22 rows=6 width=4) (actual time=0.025..0.226 rows=3 loops=1)
                                Filter: ((start_date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (start_date <= '2025-09-23 16:00:00'::timestamp without time zone))
                                Rows Removed by Filter: 2599
                          ->  Index Scan using positions_drive_id_date_index on positions p  (cost=0.43..6496.48 rows=3976 width=28) (actual time=0.012..2.349 rows=4916 loops=3)
                                Index Cond: (drive_id = d.id)
                                Filter: (car_id = '1'::smallint)
              ->  HashAggregate  (cost=85.57..86.67 rows=44 width=96) (actual time=0.185..0.239 rows=85 loops=1)
                    Group Key: (floor((EXTRACT(epoch FROM p_1.date) / '5'::numeric)) * '5'::numeric)
                    Batches: 1  Memory Usage: 88kB
                    ->  Index Scan using positions_drive_id_date_index on positions p_1  (cost=0.43..85.24 rows=44 width=48) (actual time=0.029..0.121 rows=85 loops=1)
                          Index Cond: ((drive_id IS NULL) AND (date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (date <= '2025-09-23 16:00:00'::timestamp without time zone))
                          Filter: (car_id = '1'::smallint)
Planning Time: 0.442 ms
Execution Time: 24.594 ms

cte version

Sort  (cost=40238.68..40239.18 rows=200 width=96) (actual time=28.572..28.618 rows=1054 loops=1)
"  Sort Key: ((floor((EXTRACT(epoch FROM ""*SELECT* 1"".date) / '5'::numeric)) * '5'::numeric))"
  Sort Method: quicksort  Memory: 102kB
  ->  HashAggregate  (cost=40226.04..40231.04 rows=200 width=96) (actual time=27.381..28.203 rows=1054 loops=1)
"        Group Key: (floor((EXTRACT(epoch FROM ""*SELECT* 1"".date) / '5'::numeric)) * '5'::numeric)"
        Batches: 1  Memory Usage: 849kB
        ->  Result  (cost=0.43..40086.00 rows=18672 width=48) (actual time=0.059..20.438 rows=14832 loops=1)
              ->  Append  (cost=0.43..39712.56 rows=18672 width=24) (actual time=0.051..12.072 rows=14832 loops=1)
"                    ->  Subquery Scan on ""*SELECT* 1""  (cost=0.43..39533.96 rows=18628 width=24) (actual time=0.051..11.121 rows=14747 loops=1)"
                          ->  Nested Loop  (cost=0.43..39347.68 rows=18628 width=200) (actual time=0.050..9.801 rows=14747 loops=1)
                                ->  Seq Scan on drives d  (cost=0.00..130.22 rows=6 width=4) (actual time=0.028..0.222 rows=3 loops=1)
                                      Filter: ((start_date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (start_date <= '2025-09-23 16:00:00'::timestamp without time zone))
                                      Rows Removed by Filter: 2599
                                ->  Index Scan using positions_drive_id_date_index on positions p  (cost=0.43..6496.48 rows=3976 width=28) (actual time=0.012..2.402 rows=4916 loops=3)
                                      Index Cond: (drive_id = d.id)
                                      Filter: (car_id = '1'::smallint)
"                    ->  Subquery Scan on ""*SELECT* 2""  (cost=0.43..85.24 rows=44 width=24) (actual time=0.025..0.089 rows=85 loops=1)"
                          ->  Index Scan using positions_drive_id_date_index on positions p_1  (cost=0.43..84.80 rows=44 width=200) (actual time=0.024..0.080 rows=85 loops=1)
                                Index Cond: ((drive_id IS NULL) AND (date >= '2025-09-22 16:00:00'::timestamp without time zone) AND (date <= '2025-09-23 16:00:00'::timestamp without time zone))
                                Filter: (car_id = '1'::smallint)
Planning Time: 0.698 ms
Execution Time: 28.753 ms

swiffer · 2025-10-03T16:21:39Z

perfect, thank for checking and adapting. nice outcome and ready to be merged!

…ffectively (#4964) * refactor: Optimize Grafana query for trip view * refactor: avoid double group by using CTE Signed-off-by: jaypark0006 <[email protected]> --------- Signed-off-by: jaypark0006 <[email protected]> Co-authored-by: qilei.riley <[email protected]>

…ffectively (teslamate-org#4964) * refactor: Optimize Grafana query for trip view * refactor: avoid double group by using CTE Signed-off-by: jaypark0006 <[email protected]> --------- Signed-off-by: jaypark0006 <[email protected]> Co-authored-by: qilei.riley <[email protected]>

refactor: Optimize Grafana query for trip view

c7e1ec5

JakobLichterfeld added the area:dashboard Related to a Grafana dashboard label Sep 25, 2025

JakobLichterfeld requested a review from swiffer September 25, 2025 08:16

JakobLichterfeld changed the title ~~refactor: Optimize Grafana query for trip view~~ perf: Optimize Grafana query for trip view to leverage indexes more effectively Sep 30, 2025

JakobLichterfeld added this to the v2.1.2 milestone Oct 2, 2025

refactor: avoid double group by using CTE

081cecd

Signed-off-by: jaypark0006 <[email protected]>

swiffer approved these changes Oct 3, 2025

View reviewed changes

JakobLichterfeld merged commit e604e4e into teslamate-org:main Oct 4, 2025
14 checks passed

jaypark0006 deleted the trip branch October 6, 2025 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf: Optimize Grafana query for trip view to leverage indexes more effectively #4964

perf: Optimize Grafana query for trip view to leverage indexes more effectively #4964

Uh oh!

jaypark0006 commented Sep 24, 2025 •

edited

Loading

Uh oh!

netlify bot commented Sep 24, 2025 •

edited

Loading

Uh oh!

JakobLichterfeld commented Sep 25, 2025

Uh oh!

swiffer commented Sep 27, 2025 •

edited

Loading

Uh oh!

jaypark0006 commented Oct 1, 2025

Uh oh!

jaypark0006 commented Oct 1, 2025 •

edited

Loading

Uh oh!

swiffer commented Oct 1, 2025

Uh oh!

jaypark0006 commented Oct 1, 2025

Uh oh!

jaypark0006 commented Oct 2, 2025

Uh oh!

swiffer commented Oct 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

perf: Optimize Grafana query for trip view to leverage indexes more effectively #4964

perf: Optimize Grafana query for trip view to leverage indexes more effectively #4964

Uh oh!

Conversation

jaypark0006 commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for teslamate ready!

Uh oh!

JakobLichterfeld commented Sep 25, 2025

Uh oh!

swiffer commented Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaypark0006 commented Oct 1, 2025

Uh oh!

jaypark0006 commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swiffer commented Oct 1, 2025

Uh oh!

jaypark0006 commented Oct 1, 2025

Uh oh!

jaypark0006 commented Oct 2, 2025

Uh oh!

swiffer commented Oct 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jaypark0006 commented Sep 24, 2025 •

edited

Loading

netlify bot commented Sep 24, 2025 •

edited

Loading

swiffer commented Sep 27, 2025 •

edited

Loading

jaypark0006 commented Oct 1, 2025 •

edited

Loading