[data]: url decode values in parse_hive_path by lucaschadwicklam97 · Pull Request #57625 · ray-project/ray

lucaschadwicklam97 · 2025-10-10T15:09:40Z

url decoding partition values when read into arrow table

Why are these changes needed?

PyArrow URL-encodes partition values when writing to cloud storage. To ensure the values are consistent when you read them back, this PR updates the partitioning logic to URL-decode them. See apache/arrow#34905.

Closes #57564

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run pre-commit jobs to lint the changes in this PR. (pre-commit setup)
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

gvspraveen · 2025-10-13T17:04:09Z

python/ray/data/_internal/datasource/parquet_datasource.py


    for partition_col, value in partition_col_values.items():
+        # decode the value from path
+        value = urllib.parse.unquote(value)


do you still need this change? It looks partition_col_values already gets decoded values passed from line 522, where we extract _parse_partition_column_values.
So seems like your changes in PathPartitionParser seems to be enough?

reverted. thanks @gvspraveen

iamjustinhsu · 2025-10-14T19:01:10Z

can you run ./ci/lint/lint.sh? This will make the linting tests happy

iamjustinhsu · 2025-10-14T19:04:16Z

python/ray/data/datasource/partitioning.py

-        kv_pairs = [d.split("=") for d in dirs] if dirs else []
+        kv_pairs = dict([d.split("=") for d in dirs] if dirs else [])
+        # url decode the partition values
+        kv_pairs = {k: urllib.parse.unquote(v) for k, v in kv_pairs.items()}


have u also looked into unquote_plus? It will also decode + signs in addition to what unquote supports. Just wanna make sure we covered this too.

iamjustinhsu · 2025-10-14T19:07:39Z

Thanks for the contribution! I like the changes :) One of tests is failing test_path_partition_filter_hive in https://buildkite.com/ray-project/microcheck/builds/28298#0199cee6-f2eb-42e7-9cb8-69978fe0e8c5/619-912. Can you take a look?

Signed-off-by: Lucas Lam <laml2@github.com>

…aschadwicklam97/ray into fix/urldecode_partition_vals

iamjustinhsu

👍

bveeramani

Overall LGTM. Please add a test

python/ray/data/datasource/partitioning.py

bveeramani · 2025-10-17T18:43:17Z

python/ray/data/datasource/partitioning.py

        """
        dirs = [d for d in dir_path.split("/") if d and (d.count("=") == 1)]
        kv_pairs = [d.split("=") for d in dirs] if dirs else []
+        # url decode the partition values


Could you add a test for this case?

bveeramani · 2025-10-17T18:44:31Z

I think the more holistic fix is to URL-decode when we list files from S3/GCS/HTTP, but that's a larger scope change, and I don't think we need to do it immediately

Signed-off-by: Lucas Lam <laml2@github.com>

Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com>

Signed-off-by: Lucas Lam <laml2@github.com>

…aschadwicklam97/ray into fix/urldecode_partition_vals

github-actions · 2025-11-06T00:38:27Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

lucaschadwicklam97 · 2025-11-06T16:31:40Z

@gvspraveen @iamjustinhsu any issue merging this PR? Added to existing test case and also double checked Cursor bot's recommendation to use unquote instead of unquote_plus. Its possible to have "+" literals in s3 paths, so unquote_plus("partition_value=foo+bar") == "foo bar" which is not what we want

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>

bveeramani

Made some minor refactors. LGTM

PyArrow URL-encodes partition values when writing to cloud storage. To ensure the values are consistent when you read them back, this PR updates the partitioning logic to URL-decode them. See apache/arrow#34905. Closes ray-project#57564 --------- Signed-off-by: Lucas Lam <laml2@github.com> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Lucas Lam <laml2@github.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>

PyArrow URL-encodes partition values when writing to cloud storage. To ensure the values are consistent when you read them back, this PR updates the partitioning logic to URL-decode them. See apache/arrow#34905. Closes ray-project#57564 --------- Signed-off-by: Lucas Lam <laml2@github.com> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Lucas Lam <laml2@github.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

PyArrow URL-encodes partition values when writing to cloud storage. To ensure the values are consistent when you read them back, this PR updates the partitioning logic to URL-decode them. See apache/arrow#34905. Closes ray-project#57564 --------- Signed-off-by: Lucas Lam <laml2@github.com> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Lucas Lam <laml2@github.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>

PyArrow URL-encodes partition values when writing to cloud storage. To ensure the values are consistent when you read them back, this PR updates the partitioning logic to URL-decode them. See apache/arrow#34905. Closes ray-project#57564 --------- Signed-off-by: Lucas Lam <laml2@github.com> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Lucas Lam <laml2@github.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>

PyArrow URL-encodes partition values when writing to cloud storage. To ensure the values are consistent when you read them back, this PR updates the partitioning logic to URL-decode them. See apache/arrow#34905. Closes ray-project#57564 --------- Signed-off-by: Lucas Lam <laml2@github.com> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Lucas Lam <laml2@github.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: Future-Outlier <eric901201@gmail.com>

PyArrow URL-encodes partition values when writing to cloud storage. To ensure the values are consistent when you read them back, this PR updates the partitioning logic to URL-decode them. See apache/arrow#34905. Closes ray-project#57564 --------- Signed-off-by: Lucas Lam <laml2@github.com> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Lucas Lam <laml2@github.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: peterxcli <peterxcli@gmail.com>

lucaschadwicklam97 changed the title ~~[Fix]: url decode values in parse_hive_path~~ [data]: url decode values in parse_hive_path Oct 10, 2025

lucaschadwicklam97 mentioned this pull request Oct 13, 2025

[Data] partition_cols field is url encoded but not decoded when read back #57564

Closed

gvspraveen assigned iamjustinhsu Oct 13, 2025

gvspraveen reviewed Oct 13, 2025

View reviewed changes

lucaschadwicklam97 force-pushed the fix/urldecode_partition_vals branch from a5994ca to 8545daf Compare October 13, 2025 18:01

iamjustinhsu reviewed Oct 14, 2025

View reviewed changes

Lucas Lam added 3 commits October 14, 2025 16:03

url decode values in parse_hive_path

4c433e4

Signed-off-by: Lucas Lam <laml2@github.com>

unquote plus and apply dict later

6c1a2f9

Signed-off-by: Lucas Lam <laml2@github.com>

remove earlier dict

ced3448

Signed-off-by: Lucas Lam <laml2@github.com>

lucaschadwicklam97 force-pushed the fix/urldecode_partition_vals branch from 0ddc613 to ced3448 Compare October 14, 2025 21:03

Merge branch 'master' into fix/urldecode_partition_vals

43633c8

lucaschadwicklam97 marked this pull request as ready for review October 14, 2025 21:04

lucaschadwicklam97 requested a review from a team as a code owner October 14, 2025 21:04

ray-gardener bot added data Ray Data-related issues community-contribution Contributed by the community labels Oct 15, 2025

lucaschadwicklam97 and others added 3 commits October 15, 2025 09:27

Merge branch 'master' into fix/urldecode_partition_vals

94a3f04

ruff

051aafc

Signed-off-by: Lucas Lam <laml2@github.com>

Merge branch 'fix/urldecode_partition_vals' of https://github.com/luc…

ca1ccc3

…aschadwicklam97/ray into fix/urldecode_partition_vals

lucaschadwicklam97 requested review from gvspraveen and iamjustinhsu October 15, 2025 15:55

gvspraveen approved these changes Oct 17, 2025

View reviewed changes

gvspraveen requested a review from bveeramani October 17, 2025 16:24

iamjustinhsu approved these changes Oct 17, 2025

View reviewed changes

bveeramani requested changes Oct 17, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

add test

eeffe37

Signed-off-by: Lucas Lam <laml2@github.com>

lucaschadwicklam97 force-pushed the fix/urldecode_partition_vals branch from c630ead to eeffe37 Compare October 17, 2025 21:56

Update python/ray/data/datasource/partitioning.py

1ba0645

Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: lucaschadwicklam97 <52645624+lucaschadwicklam97@users.noreply.github.com>

This comment was marked as outdated.

Sign in to view

Lucas Lam and others added 5 commits October 17, 2025 17:19

revert back to unquote

66b4b99

Signed-off-by: Lucas Lam <laml2@github.com>

Merge branch 'master' into fix/urldecode_partition_vals

9b71c76

single quotes?

a0dcef4

Signed-off-by: Lucas Lam <laml2@github.com>

Merge branch 'fix/urldecode_partition_vals' of https://github.com/luc…

f176b23

…aschadwicklam97/ray into fix/urldecode_partition_vals

Merge branch 'master' into fix/urldecode_partition_vals

8c9a342

lucaschadwicklam97 requested a review from bveeramani October 20, 2025 15:06

Merge branch 'master' into fix/urldecode_partition_vals

8c5d756

github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Nov 6, 2025

Merge branch 'master' into fix/urldecode_partition_vals

27f2224

Address review comments

7bdcb14

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>

bveeramani approved these changes Nov 6, 2025

View reviewed changes

bveeramani enabled auto-merge (squash) November 6, 2025 20:44

github-actions bot added the go add ONLY when ready to merge, run all tests label Nov 6, 2025

bveeramani merged commit 5972d1d into ray-project:master Nov 6, 2025
7 of 8 checks passed

Conversation

lucaschadwicklam97 commented Oct 10, 2025 • edited by bveeramani Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Checks

Uh oh!

gvspraveen Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

lucaschadwicklam97 Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iamjustinhsu Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

lucaschadwicklam97 Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu commented Oct 14, 2025

Uh oh!

iamjustinhsu left a comment

Choose a reason for hiding this comment

Uh oh!

bveeramani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bveeramani Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

bveeramani commented Oct 17, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions bot commented Nov 6, 2025

Uh oh!

lucaschadwicklam97 commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bveeramani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lucaschadwicklam97 commented Oct 10, 2025 •

edited by bveeramani

Loading

iamjustinhsu commented Oct 14, 2025 •

edited

Loading

lucaschadwicklam97 commented Nov 6, 2025 •

edited

Loading