Skip to content

[Data] - Update Pyarrow version to 23.0 for release tests + Update moto to 5.x.x #59489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bveeramani merged 21 commits into from
Jan 30, 2026
Merged

[Data] - Update Pyarrow version to 23.0 for release tests + Update moto to 5.x.x #59489

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5eed831
Bump PA version in release tests
alexeykudinkin Dec 16, 2025
62b21be
[Data] - Upgrade to pyarrow 22.0 + Moto 5.x
goutamvenkat-anyscale Dec 17, 2025
146fac0
Fix dep
goutamvenkat-anyscale Dec 17, 2025
15a828d
conflict
goutamvenkat-anyscale Dec 17, 2025
21dda15
more conflicts
goutamvenkat-anyscale Dec 17, 2025
e932913
update file
goutamvenkat-anyscale Dec 17, 2025
8caebe3
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Dec 17, 2025
38cfccd
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Dec 18, 2025
0e77f50
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Dec 18, 2025
63a2fec
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Dec 19, 2025
513b311
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Dec 22, 2025
50d4a7b
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Dec 23, 2025
b5d107a
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Dec 31, 2025
6c65a4e
Merge branch 'master' into pa-rel-tst-up
goutamvenkat-anyscale Jan 7, 2026
236ee25
Fix mock_aws
goutamvenkat-anyscale Jan 9, 2026
5f9c157
More test failures
goutamvenkat-anyscale Jan 9, 2026
e46aea1
Try again
goutamvenkat-anyscale Jan 10, 2026
d4194ba
idk
goutamvenkat-anyscale Jan 10, 2026
b0de7d0
Fix merge conflict
goutamvenkat-anyscale Jan 28, 2026
b3782fa
Try 23
goutamvenkat-anyscale Jan 28, 2026
7577aca
Add load_aws_managed_policies
goutamvenkat-anyscale Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .buildkite/data.rayci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ steps:
--only-tags data_non_parallel
depends_on: data9build-multipy

- label: ":database: data: arrow v21 tests"
- label: ":database: data: arrow v23 tests"
tags:
- python
- data
Expand All @@ -113,7 +113,7 @@ steps:
--except-tags data_integration,doctest,data_non_parallel,dask,needs_credentials,tensorflow_datasets
depends_on: datalbuild-multipy

- label: ":database: data: arrow v21 tests (data_non_parallel)"
- label: ":database: data: arrow v23 tests (data_non_parallel)"
tags:
- python
- data
Expand All @@ -128,7 +128,7 @@ steps:
--only-tags data_non_parallel
depends_on: datalbuild-multipy

- label: ":database: data: arrow v21 py{{matrix.python}} tests ({{matrix.worker_id}})"
- label: ":database: data: arrow v23 py{{matrix.python}} tests ({{matrix.worker_id}})"
key: datal_python_tests
if: build.pull_request.labels includes "continuous-build" || pipeline.id == "0189e759-8c96-4302-b6b5-b4274406bf89" || pipeline.id == "018f4f1e-1b73-4906-9802-92422e3badaa"
tags:
Expand All @@ -145,7 +145,7 @@ steps:
python: ["3.12"]
worker_id: ["0", "1"]

- label: ":database: data: arrow v21 py{{matrix.python}} tests (data_non_parallel)"
- label: ":database: data: arrow v23 py{{matrix.python}} tests (data_non_parallel)"
key: datal_python_non_parallel_tests
if: build.pull_request.labels includes "continuous-build" || pipeline.id == "0189e759-8c96-4302-b6b5-b4274406bf89" || pipeline.id == "018f4f1e-1b73-4906-9802-92422e3badaa"
tags:
Expand Down
2 changes: 1 addition & 1 deletion ci/docker/datal.build.wanda.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ srcs:
- python/requirements/ml/data-test-requirements.txt
build_args:
- DOCKER_IMAGE_BASE_BUILD=cr.ray.io/rayproject/oss-ci-base_ml-py$PYTHON
- ARROW_VERSION=21.*
- ARROW_VERSION=23.*
tags:
- cr.ray.io/rayproject/databuild-py$PYTHON
4 changes: 3 additions & 1 deletion python/ray/data/tests/mock_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,9 @@ def start_service(service_name, host, port):
# Always use port conflict resolution to be safe
port = _find_available_port(host, port)

args = [moto_svr_path, service_name, "-H", host, "-p", str(port)]
# moto 5.x no longer accepts a service name argument - all services
# are served on a single endpoint
args = [moto_svr_path, "-H", host, "-p", str(port)]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

With this change, the service_name parameter is no longer used to configure the moto_server. It's only used in error messages on lines 81 and 91. Since moto v5 serves all services on a single endpoint, the concept of starting a single service is obsolete. Consider removing the service_name parameter from the function signature and updating the error messages to something more generic like "Can not start moto server". This would require updating the call site on line 124 as well.

# For debugging
# args = '{0} {1} -H {2} -p {3} 2>&1 | \
# tee -a /tmp/moto.log'.format(moto_svr_path, service_name, host, port)
Expand Down
4 changes: 3 additions & 1 deletion python/ray/tests/mock_s3_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@

def start_service(service_name, host, port):
moto_svr_path = shutil.which("moto_server")
args = [moto_svr_path, service_name, "-H", host, "-p", str(port)]
# moto 5.x no longer accepts a service name argument - all services
# are served on a single endpoint
args = [moto_svr_path, "-H", host, "-p", str(port)]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

With this change, the service_name parameter is no longer used to configure moto_server. It is only used in an error message on line 36. Since moto v5 serves all services on a single endpoint, the parameter is misleading. It would be best to remove service_name from the function signature and update the error message to be more generic (e.g., "Can not start moto server"). This would also require updating the call sites on lines 70, 80, and 90.

process = sp.Popen(
args, stdin=sp.PIPE, stdout=sp.DEVNULL, stderr=sp.DEVNULL
) # shell=True
Expand Down
52 changes: 20 additions & 32 deletions python/ray/tests/test_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,10 @@
from unittest import mock
from unittest.mock import MagicMock, patch

import moto
import pytest
import yaml
from click.testing import CliRunner
from moto import mock_ec2, mock_iam
from moto import mock_aws
from testfixtures import Replacer
from testfixtures.popen import MockPopen, PopenBehaviour

Expand Down Expand Up @@ -113,18 +112,25 @@ def configure_aws():
os.environ["AWS_SESSION_TOKEN"] = "testing"

# moto (boto3 mock) only allows a hardcoded set of AMIs
dlami = (
moto.ec2.models.ec2_backends["us-west-2"]["us-west-2"]
.describe_images(filters={"name": "Deep Learning AMI Ubuntu*"})[0]
.id
)
aws_config.DEFAULT_AMI["us-west-2"] = dlami
list_instances_mock = MagicMock(return_value=boto3_list)
with patch(
"ray.autoscaler._private.aws.node_provider.list_ec2_instances",
list_instances_mock,
):
yield
# Use mock_aws context manager and boto3 to find the AMI
import boto3

# In moto 5.x, AWS managed policies (e.g., AmazonEC2FullAccess) are not
# loaded by default for performance. Enable them since the autoscaler
# attaches these policies to the IAM role.
with mock_aws(config={"iam": {"load_aws_managed_policies": True}}):
ec2_client = boto3.client("ec2", region_name="us-west-2")
images = ec2_client.describe_images(
Filters=[{"Name": "name", "Values": ["Deep Learning AMI Ubuntu*"]}]
)["Images"]
dlami = images[0]["ImageId"]
aws_config.DEFAULT_AMI["us-west-2"] = dlami
list_instances_mock = MagicMock(return_value=boto3_list)
with patch(
"ray.autoscaler._private.aws.node_provider.list_ec2_instances",
list_instances_mock,
):
yield


@pytest.fixture(scope="function")
Expand Down Expand Up @@ -636,8 +642,6 @@ def test_ray_start_block_and_stop(configure_lang, monkeypatch, tmp_path, cleanup
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_up(
configure_lang, _unlink_test_ssh_key, configure_aws, monkeypatch, tmp_path
):
Expand Down Expand Up @@ -677,8 +681,6 @@ def commands_mock(command, stdin):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_up_docker(
configure_lang, _unlink_test_ssh_key, configure_aws, monkeypatch, tmp_path
):
Expand Down Expand Up @@ -720,8 +722,6 @@ def commands_mock(command, stdin):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_up_record(
configure_lang, _unlink_test_ssh_key, configure_aws, monkeypatch, tmp_path
):
Expand Down Expand Up @@ -754,8 +754,6 @@ def commands_mock(command, stdin):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_attach(configure_lang, configure_aws, _unlink_test_ssh_key):
def commands_mock(command, stdin):
# TODO(maximsmol): this is a hack since stdout=sys.stdout
Expand Down Expand Up @@ -796,8 +794,6 @@ def commands_mock(command, stdin):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_attach_with_ip(configure_lang, configure_aws, _unlink_test_ssh_key):
from ray.autoscaler._private.commands import get_worker_node_ips

Expand Down Expand Up @@ -876,8 +872,6 @@ def commands_verifier(calls):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_dashboard(configure_lang, configure_aws, _unlink_test_ssh_key):
def commands_mock(command, stdin):
# TODO(maximsmol): this is a hack since stdout=sys.stdout
Expand Down Expand Up @@ -910,8 +904,6 @@ def commands_mock(command, stdin):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_exec(configure_lang, configure_aws, _unlink_test_ssh_key):
def commands_mock(command, stdin):
# TODO(maximsmol): this is a hack since stdout=sys.stdout
Expand Down Expand Up @@ -963,8 +955,6 @@ def commands_verifier(calls):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_submit(configure_lang, configure_aws, _unlink_test_ssh_key):
def commands_mock(command, stdin):
# TODO(maximsmol): this is a hack since stdout=sys.stdout
Expand Down Expand Up @@ -1355,8 +1345,6 @@ def test_ray_drain_node(monkeypatch):
sys.platform == "darwin" and "travis" in os.environ.get("USER", ""),
reason=("Mac builds don't provide proper locale support"),
)
@mock_ec2
@mock_iam
def test_ray_cluster_dump(configure_lang, configure_aws, _unlink_test_ssh_key):
def commands_mock(command, stdin):
print("This is a test!")
Expand Down
2 changes: 1 addition & 1 deletion python/requirements/ml/data-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ modin==0.22.2; python_version < '3.12'
pandas==1.5.3; python_version < '3.12'
modin==0.31.0; python_version >= '3.12'
pandas==2.2.2; python_version >= '3.12'
responses==0.13.4
responses>=0.15.0
pymars>=0.8.3; python_version < "3.12"
2 changes: 1 addition & 1 deletion python/requirements/test-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jsonpatch==1.32
kubernetes==24.2.0
llvmlite==0.42.0
lxml>=6.0.2
moto[s3,server]==4.2.12
moto[s3,server]==5.1.18
Copy link
Collaborator

@aslonnie aslonnie Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please update the related dependency set.

cc @elliot-barn

Copy link
Contributor

@elliot-barn elliot-barn Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea you don't need to update requirements_compiled.txt manually. You can run these scripts in python 3.11 environment ci/ci.sh compile_pip_dependencies && bazel run //ci/raydepsets:raydepsets -- build --all-configs

Copy link
Contributor

@alexeykudinkin alexeykudinkin Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold on, @goutamvenkat-anyscale would new moto be compatible w/ PA < 22?

Copy link
Contributor Author

@goutamvenkat-anyscale goutamvenkat-anyscale Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's compatible.
Tested with this script:

import boto3
from moto import mock_aws

@mock_aws
def test_s3_put_get():
    s3 = boto3.client("s3", region_name="us-east-1")
    s3.create_bucket(Bucket="test-bucket")
    
    # Test basic put/get
    s3.put_object(Bucket="test-bucket", Key="test.txt", Body=b"spam")
    resp = s3.get_object(Bucket="test-bucket", Key="test.txt")
    body = resp["Body"].read()
    
    print(f"Expected: b'spam'")
    print(f"Got:      {body!r}")
    assert body == b"spam", f"Mismatch! Got {body!r}"
    print("✓ Basic test passed!")
    
    # Test with larger payload (more likely to trigger chunked encoding)
    large_body = b"x" * 10_000
    s3.put_object(Bucket="test-bucket", Key="large.bin", Body=large_body)
    resp = s3.get_object(Bucket="test-bucket", Key="large.bin")
    result = resp["Body"].read()
    
    assert result == large_body, f"Large payload mismatch! Got {len(result)} bytes"
    print("✓ Large payload test passed!")

if __name__ == "__main__":
    import pyarrow
    import moto
    print(f"pyarrow: {pyarrow.__version__}")
    print(f"moto:    {moto.__version__}")
    print()
    test_s3_put_get()

Output:

pyarrow: 21.0.0
moto:    5.1.20

Expected: b'spam'
Got:      b'spam'

mypy==1.7.0
numba==0.59.1
openpyxl==3.0.10
Expand Down
32 changes: 12 additions & 20 deletions python/requirements_compiled.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ annotated-types==0.6.0
antlr4-python3-runtime==4.11.1
# via
# fugue-sql-antlr
# moto
# qpd
anyio==4.12.0
# via
Expand Down Expand Up @@ -392,14 +393,13 @@ cryptography==44.0.3
# azure-cli-core
# azure-identity
# azure-storage-blob
# joserfc
# moto
# msal
# paramiko
# pyjwt
# pyopenssl
# python-jose
# snowflake-connector-python
# sshpubkeys
# trustme
cupy-cuda12x==13.4.0 ; sys_platform != "darwin"
# via
Expand Down Expand Up @@ -484,11 +484,6 @@ docutils==0.19
# sphinx
dulwich==0.21.6
# via comet-ml
ecdsa==0.18.0
# via
# moto
# python-jose
# sshpubkeys
entrypoints==0.4
# via nbconvert
et-xmlfile==1.1.0
Expand Down Expand Up @@ -889,17 +884,19 @@ joblib==1.2.0
# via
# -r python/requirements/test-requirements.txt
# scikit-learn
joserfc==1.5.0
# via moto
jschema-to-python==1.2.3
# via cfn-lint
json5==0.9.14
# via jupyterlab-server
jsondiff==2.0.0
# via moto
jsonpatch==1.32
# via
# -r python/requirements/cloud-requirements.txt
# -r python/requirements/test-requirements.txt
# cfn-lint
jsonpath-ng==1.7.0
# via moto
jsonpickle==3.0.2
# via jschema-to-python
jsonpointer==2.4
Expand Down Expand Up @@ -1089,7 +1086,7 @@ more-itertools==10.7.0
# via configspace
mosaicml==0.3.1 ; python_version < "3.12"
# via -r python/requirements/ml/train-test-requirements.txt
moto==4.2.12
moto==5.1.18
Copy link
Collaborator

@aslonnie aslonnie Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elliot-barn , could you help update python3.13 requirements to be consistent with the changes in this PR too?

# via -r python/requirements/test-requirements.txt
moviepy==0.2.3.1
# via -r python/requirements/ml/rllib-test-requirements.txt
Expand Down Expand Up @@ -1497,6 +1494,8 @@ plotly==5.23.0
# via ax-platform
pluggy==1.3.0
# via pytest
ply==3.11
# via jsonpath-ng
polars==1.36.1
# via -r python/requirements/test-requirements.txt
polars-runtime-32==1.36.1
Expand Down Expand Up @@ -1579,7 +1578,7 @@ py==1.11.0
# via pytest-forked
py-cpuinfo==9.0.0
# via deepspeed
py-partiql-parser==0.5.0
py-partiql-parser==0.6.3
# via moto
py-spy==0.4.0 ; python_version < "3.12"
# via -r python/requirements.txt
Expand All @@ -1605,7 +1604,6 @@ pyasn1==0.5.1
# via
# oauth2client
# pyasn1-modules
# python-jose
# rsa
pyasn1-modules==0.3.0
# via
Expand Down Expand Up @@ -1786,8 +1784,6 @@ python-dateutil==2.8.2
# strictyaml
python-dotenv==1.2.1
# via testcontainers
python-jose==3.3.0
# via moto
python-json-logger==2.0.7
# via jupyter-events
python-lsp-jsonrpc==1.0.0
Expand Down Expand Up @@ -1842,6 +1838,7 @@ pyyaml==6.0.3
# pymars
# pytorch-lightning
# ray
# responses
# timm
# transformers
# wandb
Expand Down Expand Up @@ -1926,7 +1923,7 @@ requests-oauthlib==2.0.0
# msrest
requests-toolbelt==1.0.0
# via comet-ml
responses==0.13.4
responses==0.25.8
# via
# -r python/requirements/ml/data-requirements.txt
# moto
Expand Down Expand Up @@ -1965,7 +1962,6 @@ rsa==4.7.2
# gcs-oauth2-boto-plugin
# google-auth
# oauth2client
# python-jose
ruamel-yaml==0.17.40
# via
# semgrep
Expand Down Expand Up @@ -2058,7 +2054,6 @@ six==1.16.0
# azure-core
# bleach
# configobj
# ecdsa
# fs
# gcs-oauth2-boto-plugin
# google-apitools
Expand All @@ -2080,7 +2075,6 @@ six==1.16.0
# python-dateutil
# pyu2f
# pyvmomi
# responses
# rfc3339-validator
# tensorboard
# tensorflow
Expand Down Expand Up @@ -2139,8 +2133,6 @@ sqlglot==25.6.1
# via fugue
sqlparse==0.5.1
# via mlflow-skinny
sshpubkeys==3.3.1
# via moto
stack-data==0.6.3
# via ipython
stanio==0.3.0
Expand Down