ES|QL Support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX #125143

craigtaverner · 2025-03-18T18:00:54Z

In support of the feature request at #123903

Checklist:

Prototype
- Implement basic ST_GEOHASH on points
- Add support for optional bounds parameter
- Support multivalues (incoming and outgoing)
- Add support for ST_GEOTILE
- Add support for ST_GEOHEX
Productize:
- Use bounded predicates to filter by tile, not geometry
  - Use GeoTileBoundedPredicate for correct geotile bounds filtering
  - Use GeohashBoundedPredicate for correct geohash bounds filtering
  - Copy GeoHexCellIdSource.GeoHexPredicate and H3SphericalUtil classes into esql module and use them for correct H3 bounds filtering
- Refine API
  - Change functions to return long grid-ids instead of keyword for performance and add new functions for keyword alternatives
  - Create conversion functions between long and keyword versions of the grid-ids
  - Create functions to generate geometries from the grid-ids
- Additional refinements
  - Add unit tests and generate docs
  - Tests for geo_grid queries (| WHERE ST_GEOHEX(location, 1) == ST_GEOHEX_TO_LONG("812bbffffffffff"))
  - Check input parameters for validity (eg. negative precision)
  - Errors tests
  - Precision range validation
  - Remove support for _TO_GEOSHAPE functions
  - Remove support for dynamic precision (from field or calculated values)
  - Test coverage of bounds (and get bounds types into the docs)
  - Restrict bounds parameter to only geo_shape of an envelope (no hidden envelope calculation)
  - Test that prove docs-values code path works (ie. function before spatial stats)
  - Add license checks for geohex grid
  - Docs links to query dsl
Extra features (could be done here or in later PRs)
- Bring back support for _TO_GEOSHAPE functions, including enhanced WKB (include grid-id)
- Lucene pushdown for geo_grid queries (see GeoGridQueryBuilder, and copy to ES|QL)
- Expand support to geo_shape fields (including license checks)
- Add pushdown to Lucene for grid aggregations
- Bring back support for dynamic precision (from field or calculated values)
- Non-foldable bounds

iverase · 2025-03-19T19:43:08Z

...src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/spatial/StGeotile.java

+    }
+
+    protected static BytesRef calculateGeotile(Point point, int precision, Rectangle bounds) {
+        // For points, filtering the point is as good as filtering the tile


I don't think this statement is correct. See https://github.com/elastic/elasticsearch/blob/368d47131f2d74742ae89005680352389387bdb8/server/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoTileBoundedPredicate.java

I agree. The current code was a quick-and dirty placeholder for the draft PR. I'll definitely switch over to the better code you suggest.

iverase · 2025-04-09T15:46:39Z

I was trying queries like:

ROW location=TO_GEOPOINT("POINT(0 0)")| EVAL hex=ST_GEOHEX(location, -1)

In case of geohex and geotile we provide a good error although looking into the stack trace it feels to me that we are not checking the input while parsing and we only fail when trying to apply the function to the point. More over, for geohash it does not throw an error.

Let's improve the checking of input parameters.

This is because we want to move the ST_GEOXXX functions to produce longs only, for performance reasons, and need the convenience functions to display these either as strings or as actual shapes. This is both useful to the users, as well as easier to do manual tests where we visualize the results on a map.

This involved copying the H3 utils from spatial plugin to esql. We should rather find a common location or dependency for these.

…duplicated code.

And refine tests to better cover bounds parameter, fixing missing type information in the docs

iverase · 2025-05-29T07:07:06Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial-grid.csv-spec

+FROM airports
+| EVAL points = ["POINT(0.0 30.0)", "POINT(12.0 60.0)"]
+| EVAL geohash = ST_GEOHASH(location, 2, TO_GEOPOINT(points))
+| WHERE geohash IS NOT NULL


Why is this needed?

I think I understand now. It is strange how the bounds are defined here.

Support for bounds defined by points is now removed.

Now only geo_shape bounds are supported

Also enhance tests to validate against actual bounds

iverase

LGTM

I think we need to strength the test coverage for multi-value case but we can do it in a follow up

elasticsearchmachine · 2025-06-03T09:50:46Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 125143

Added support for the three primary scalar grid functions: * `ST_GEOHASH(geom, precision)` * `ST_GEOTILE(geom, precision)` * `ST_GEOHEX(geom, precision)` As well as versions of these three that take an optional `geo_shape` boundary (must be a `BBOX` ie. `Rectangle`). And also supporting conversion functions that convert the grid-id from long to string and back to long. This work represents the core of the feature to support geo-grid aggregations in ES|QL.

… (#128938) * ES|QL Support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX (#125143) Added support for the three primary scalar grid functions: * `ST_GEOHASH(geom, precision)` * `ST_GEOTILE(geom, precision)` * `ST_GEOHEX(geom, precision)` As well as versions of these three that take an optional `geo_shape` boundary (must be a `BBOX` ie. `Rectangle`). And also supporting conversion functions that convert the grid-id from long to string and back to long. This work represents the core of the feature to support geo-grid aggregations in ES|QL. * Remove license checking mechanism from tests (unsupported in 8.x) * Add 8.x level generated docs * Link new spatial functions into the docs * Remove unused imports after removal of license level tests * Fix list syntax * Fix asciidoc links

Added support for the three primary scalar grid functions: * `ST_GEOHASH(geom, precision)` * `ST_GEOTILE(geom, precision)` * `ST_GEOHEX(geom, precision)` As well as versions of these three that take an optional `geo_shape` boundary (must be a `BBOX` ie. `Rectangle`). And also supporting conversion functions that convert the grid-id from long to string and back to long. This work represents the core of the feature to support geo-grid aggregations in ES|QL.

#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in #129581 are likely better. As a first step, rmove the spatial grid functions added in #125143 from release builds so they don't get released into 8.19/9.1. --------- Co-authored-by: Craig Taverner <[email protected]>

elastic#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in elastic#129581 are likely better. As a first step, rmove the spatial grid functions added in elastic#125143 from release builds so they don't get released into 8.19/9.1. --------- Co-authored-by: Craig Taverner <[email protected]> (cherry picked from commit efb1397) # Conflicts: # docs/reference/query-languages/esql/_snippets/lists/spatial-functions.md # docs/reference/query-languages/esql/functions-operators/spatial-functions.md

#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in #129581 are likely better. As a first step, remove the spatial grid functions added in #125143 from release builds so they don't get released into 8.19/9.1. --------- Co-authored-by: Craig Taverner <[email protected]> (cherry picked from commit efb1397) # Conflicts: # docs/reference/query-languages/esql/_snippets/lists/spatial-functions.md # docs/reference/query-languages/esql/functions-operators/spatial-functions.md

elastic#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in elastic#129581 are likely better. As a first step, rmove the spatial grid functions added in elastic#125143 from release builds so they don't get released into 8.19/9.1. --------- Co-authored-by: Craig Taverner <[email protected]>

github-actions bot deployed to docs-preview March 18, 2025 18:01 View deployment

elasticsearchmachine added the v9.1.0 label Mar 18, 2025

craigtaverner force-pushed the esql_grid_aggs branch from e3b2faf to 5c3d0ee Compare March 18, 2025 18:20

github-actions bot deployed to docs-preview March 18, 2025 18:20 View deployment

craigtaverner force-pushed the esql_grid_aggs branch from 5c3d0ee to 2e7e642 Compare March 19, 2025 10:19

github-actions bot deployed to docs-preview March 19, 2025 10:19 View deployment

iverase reviewed Mar 19, 2025

View reviewed changes

craigtaverner force-pushed the esql_grid_aggs branch from 2e7e642 to afd6a94 Compare March 20, 2025 10:53

craigtaverner force-pushed the esql_grid_aggs branch from bcd5ec4 to 9f851d1 Compare April 8, 2025 12:46

craigtaverner force-pushed the esql_grid_aggs branch from 9f851d1 to e362e57 Compare April 15, 2025 11:06

github-actions bot deployed to docs-preview April 15, 2025 15:58 View deployment

craigtaverner force-pushed the esql_grid_aggs branch 3 times, most recently from eadd44d to ff82ace Compare April 17, 2025 09:05

craigtaverner added 15 commits April 19, 2025 15:23

Add initial geohash function to ESQL

ab3d9d6

Add generated files

89559f6

Add support for optional bounds parameter

dd346c2

Added geotile and geohex

a4db2a3

Add ESQL Capability and reduce test flakiness

da13b47

Fix failing EsqlNodeSubclassTests from incorrect NodeInfo

9c83b64

Use GeoTileBoundedPredicate for geotile bounds filtering

04b2c89

Convert geogrid functions to produce only long values

5ed543b

Fixes after rebasing on main

f3a19a0

Support geo-bounded-predicate for geohash and geohex

7e1989e

This involved copying the H3 utils from spatial plugin to esql. We should rather find a common location or dependency for these.

Added unit tests for new functions and resulting docs

6d115b4

Output POLYGON instead of BBOX for TO_GEOSHAPE functions

983b977

Make test less flaky

0debf2e

Use wrapper classes on the bounded predicates to simplify and reduce …

a0c1e09

…duplicated code.

craigtaverner added 3 commits May 21, 2025 18:35

Merge branch 'main' into esql_grid_aggs

71b6b33

Fix functions count after removing to_geoshape methods

ea324e4

Remove all field evaluators except spatial field evaluator

aa36091

And refine tests to better cover bounds parameter, fixing missing type information in the docs

iverase reviewed May 29, 2025

View reviewed changes

craigtaverner added 5 commits May 30, 2025 10:45

Remove support for geo_point defined bounds

8af276b

Now only geo_shape bounds are supported

Disallow any geo_shape bounds other than BBOX

9cca2b2

Also enhance tests to validate against actual bounds

Merge remote-tracking branch 'origin/main' into esql_grid_aggs

5268880

Add license checks and integration tests to test license checks

bcb07b2

Merge remote-tracking branch 'origin/main' into esql_grid_aggs

f139ac0

iverase approved these changes Jun 2, 2025

View reviewed changes

craigtaverner added 4 commits June 2, 2025 14:18

Fix license checks in function unit tests

0f90ddb

Experiment to reduce serverless flakiness

7de66e7

Merge branch 'main' into esql_grid_aggs

620df9d

Merge branch 'main' into esql_grid_aggs

610c99f

craigtaverner added v8.19.0 auto-backport Automatically create backport pull requests when merged labels Jun 3, 2025

craigtaverner merged commit 11f0c55 into elastic:main Jun 3, 2025
18 checks passed

elasticsearchmachine added the backport pending label Jun 3, 2025

craigtaverner mentioned this pull request Jun 4, 2025

[8.19] ES|QL Support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX (#125143) #128938

Merged

craigtaverner mentioned this pull request Jun 17, 2025

Support geohash, geotile and geohex grid types #129581

Open

alex-spies mentioned this pull request Jun 23, 2025

ESQL: Hide spatial grid functions behind SNAPSHOT #129839

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ES|QL Support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX #125143

ES|QL Support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX #125143

Uh oh!

craigtaverner commented Mar 18, 2025 •

edited

Loading

Uh oh!

iverase Mar 19, 2025

Uh oh!

craigtaverner Mar 20, 2025

Uh oh!

iverase commented Apr 9, 2025 •

edited

Loading

Uh oh!

iverase May 29, 2025

Uh oh!

iverase May 29, 2025

Uh oh!

craigtaverner Jun 2, 2025

Uh oh!

iverase left a comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 3, 2025

Uh oh!

Uh oh!

ES|QL Support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX #125143

ES|QL Support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX #125143

Uh oh!

Conversation

craigtaverner commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iverase Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

craigtaverner Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

iverase commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iverase May 29, 2025

Choose a reason for hiding this comment

Uh oh!

iverase May 29, 2025

Choose a reason for hiding this comment

Uh oh!

craigtaverner Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

iverase left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 3, 2025

💔 Backport failed

Uh oh!

Uh oh!

craigtaverner commented Mar 18, 2025 •

edited

Loading

iverase commented Apr 9, 2025 •

edited

Loading