[Data] Compute Expressions-datetime

[400Ping]

Description

Completing the datetime namespace operations

Related issues

Related to #58674

Additional information

[gemini-code-assist]

Code Review

This pull request introduces datetime expression operations under the .dt namespace, which is a great addition. The implementation in dt_namespace.py is mostly clean and follows existing patterns.

However, there is a critical omission: the dt property is not defined on the Expr class in python/ray/data/expressions.py. Without this, expressions like col('my_date').dt.year() will fail. This property needs to be added, similar to how .list, .str, and .struct are implemented. You can add the following property to the Expr class:

    @property
    def dt(self) -> "_DatetimeNamespace":
        """Access datetime operations for this expression."""
        from ray.data.namespace_expressions.dt_namespace import _DatetimeNamespace

        return _DatetimeNamespace(self)

I've also left a couple of comments in dt_namespace.py regarding potential improvements:

• A high-severity issue with how the return data type is determined for ceil, floor, and round functions, which could lead to incorrect type inference.
• A medium-severity suggestion to refactor duplicated code for better maintainability.

[goutamvenkat-anyscale]

lgtm!

[goutamvenkat-anyscale]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request significantly enhances Ray Data's expression capabilities by adding a dedicated datetime namespace. It allows users to perform a wide range of temporal manipulations directly on columns, such as extracting date parts, formatting, and rounding timestamps. This improves the expressiveness and convenience for data processing workflows involving datetime data.

Highlights

• New Datetime Namespace: Introduced a new _DatetimeNamespace for Ray Data expressions, enabling direct datetime operations on columns using the .dt accessor (e.g., col('timestamp').dt.year()).
• Comprehensive Datetime Operations: The new namespace includes methods for extracting temporal components (year, month, day, hour, minute, second), formatting datetimes with strftime, and temporal rounding operations (ceil, floor, round).
• Integration with Expression API: The dt accessor is now a property of the Expr class, making datetime operations seamlessly available within the Ray Data expression framework.
• New Test Coverage: Added new tests to validate the functionality of all datetime namespace operations and to ensure proper error handling when used on non-datetime column types.

Changelog

• python/ray/data/expressions.py
  • Added _DatetimeNamespace to TYPE_CHECKING imports.
  • Introduced a dt property to the Expr class to provide access to datetime operations.
  • Included _DatetimeNamespace in the __all__ list and the __getattr__ method for lazy loading.
• python/ray/data/namespace_expressions/dt_namespace.py
  • Added a new file defining the _DatetimeNamespace class.
  • Implemented methods for extracting year, month, day, hour, minute, and second components.
  • Added strftime method for custom datetime formatting.
  • Implemented ceil, floor, and round methods for temporal rounding, preserving the input data type.
  • Utilized pyarrow_udf for all datetime operations.
• python/ray/data/tests/test_namespace_expressions.py
  • Imported the datetime module for testing purposes.
  • Added test_datetime_namespace_all_operations to comprehensively test all new datetime functions.
  • Added test_dt_namespace_invalid_dtype_raises to verify that using the dt namespace on non-datetime columns raises an exception.
  • Made minor formatting adjustments in existing struct tests.

Activity

• A cursor[bot] comment highlighted that the _DatetimeNamespace was implemented but inaccessible due to a missing dt property on the Expr class.
• A gemini-code-assist[bot] comment suggested using DataType.temporal() for return_dtype in temporal rounding functions to avoid object type inference issues, which was acknowledged by goutamvenkat-anyscale.
• A gemini-code-assist[bot] comment recommended refactoring the year, month, etc., methods to reduce code duplication using a helper function, also acknowledged by goutamvenkat-anyscale.
• goutamvenkat-anyscale inquired about the tie_breaker parameter for round_temporal and suggested using Literal for unit parameters in temporal functions.
• goutamvenkat-anyscale requested adding comprehensive tests for the new datetime functionality in test_namespace_expressions.py, including a consolidated test and a failure case test.
• A cursor[bot] comment pointed out that temporal rounding should preserve the specific input temporal type (self._expr.data_type) rather than using a generic DataType.temporal(), which goutamvenkat-anyscale confirmed with an example.
• goutamvenkat-anyscale suggested using the rows_same utility for assertions in tests.
• A cursor[bot] comment identified a low-severity bug regarding a test parametrization mismatch.