Passing a POSIXct to a polars DataFrame can change some components?

[etiennebacher]

I don't remember if this is known or not, and I don't have much experience with POSIXct. @eitsupi is this a bug?

date <- as.POSIXct("2493-05-15 00:00:00 CEST")

data.frame(x1 = date)
#>           x1
#> 1 2493-05-15  <<<<<<<<<<<<<< day is 15
polars::pl$DataFrame(x1 = date)
#> shape: (1, 1)
#> ┌─────────────────────┐
#> │ x1                  │
#> │ ---                 │
#> │ datetime[ms]        │
#> ╞═════════════════════╡
#> │ 2493-05-14 23:00:00 │ <<<<<<<<<<<<<< day is 14
#> └─────────────────────┘
neopolars::pl$DataFrame(x1 = date)
#> shape: (1, 1)
#> ┌─────────────────────┐
#> │ x1                  │
#> │ ---                 │
#> │ datetime[ms]        │
#> ╞═════════════════════╡
#> │ 2493-05-14 23:00:00 │ <<<<<<<<<<<<<< day is 14
#> └─────────────────────┘

---

Noticed in etiennebacher/tidypolars#170 (comment)

[eitsupi]

Is it possible that daylight saving time is affected? What happens in other years?
What is the system timezone?

[eitsupi]

Just for the record, CEST is irrelevant.

identical(
  as.POSIXct("2493-05-15 00:00:00 CEST"),
  as.POSIXct("2493-05-15 00:00:00")
)
#> [1] TRUE

^{Created on 2025-03-31 with reprex v2.1.1}

What is relevant is the time zone setting of the system.

withr::with_timezone("Asia/Tokyo", {
  date <- as.POSIXct("2493-05-15 00:00:00 CEST")
  neopolars::pl$DataFrame(x1 = date)
})
#> shape: (1, 1)
#> ┌─────────────────────┐
#> │ x1                  │
#> │ ---                 │
#> │ datetime[ms]        │
#> ╞═════════════════════╡
#> │ 2493-05-15 00:00:00 │
#> └─────────────────────┘

withr::with_timezone("Europe/Amsterdam", {
  date <- as.POSIXct("2493-05-15 00:00:00 CEST")
  neopolars::pl$DataFrame(x1 = date)
})
#> shape: (1, 1)
#> ┌─────────────────────┐
#> │ x1                  │
#> │ ---                 │
#> │ datetime[ms]        │
#> ╞═════════════════════╡
#> │ 2493-05-14 23:00:00 │
#> └─────────────────────┘

Do you think this is related to daylight savings time? (I know daylight saving time could have been eliminated in year 2493 :))

[etiennebacher]

Do you think this is related to daylight savings time? (I know daylight saving time could have been eliminated in year 2493 :))

Probably, this is the earliest I could come up with:

withr::with_timezone("Europe/Luxembourg", {
  date <- as.POSIXct("2100-03-29 00:00:00")
  neopolars::pl$DataFrame(x1 = date)
})
#> shape: (1, 1)
#> ┌─────────────────────┐
#> │ x1                  │
#> │ ---                 │
#> │ datetime[ms]        │
#> ╞═════════════════════╡
#> │ 2100-03-28 23:00:00 │
#> └─────────────────────┘

and usually DST changes in CET happen around this date (we had one yesterday).

But why this doesn't happen before 2100, that's a mystery.

[eitsupi]

But why this doesn't happen before 2100, that's a mystery.

I do not know the structure of the time zone database, but presumably the database should record in which year daylight savings time runs.
Of course, accuracy should not be required with respect to the far future.

So there is nothing we can do here. Just remove the test case.

[etiennebacher]

It seems that py-polars doesn't have this though:

• R

date <- as.POSIXct("2100-03-29 00:00:00", tz = "Europe/Luxembourg")
neopolars::pl$DataFrame(x1 = date)
#> shape: (1, 1)
#> ┌─────────────────────────────────┐
#> │ x1                              │
#> │ ---                             │
#> │ datetime[ms, Europe/Luxembourg] │
#> ╞═════════════════════════════════╡
#> │ 2100-03-28 23:00:00 CET         │
#> └─────────────────────────────────┘

• Python

import polars as pl

print(
    pl.DataFrame({"x1": ["2100-03-29 00:00:00"]}).select(
        pl.col("x1").str.strptime(
            pl.Datetime("ms", time_zone="Europe/Luxembourg"), "%Y-%m-%d %H:%M:%S"
        )
    )
)
shape: (1, 1)
┌─────────────────────────────────┐
│ x1                              │
│ ---                             │
│ datetime[ms, Europe/Luxembourg] │
╞═════════════════════════════════╡
│ 2100-03-29 00:00:00 CET         │
└─────────────────────────────────┘

[etiennebacher]

And just to add more confusion:

posixct <- as.POSIXct("2100-03-29 00:00:00", tz = "Europe/Luxembourg")
posixlt <- as.POSIXlt("2100-03-29 00:00:00", tz = "Europe/Luxembourg")
neopolars::pl$DataFrame(x1 = posixct)
#> shape: (1, 1)
#> ┌─────────────────────────────────┐
#> │ x1                              │
#> │ ---                             │
#> │ datetime[ms, Europe/Luxembourg] │
#> ╞═════════════════════════════════╡
#> │ 2100-03-28 23:00:00 CET         │
#> └─────────────────────────────────┘
neopolars::pl$DataFrame(x1 = posixlt)
#> shape: (1, 1)
#> ┌─────────────────────────────────┐
#> │ x1                              │
#> │ ---                             │
#> │ datetime[ns, Europe/Luxembourg] │
#> ╞═════════════════════════════════╡
#> │ 2100-03-29 00:00:00 CET         │
#> └─────────────────────────────────┘

[eitsupi]

POSIXct and POSIXlt have completely different internal data storage strategies: POSIXct holds the number of seconds elapsed since the POSIX origin, whereas POSIXlt holds the numerical value of each component (and thus may even hold invalid times).

[eitsupi]

It seems that py-polars doesn't have this though:

It is not. The problem here is that the internal representation of 2100-03-29 00:00:00 Europe/Luxembourg on the R side is 4109954400.
However, this is not the case within Polars.

as.POSIXct("2100-03-29 00:00:00", tz = "Europe/Luxembourg") |> unclass()
#> [1] 4109954400
#> attr(,"tzone")
#> [1] "Europe/Luxembourg"

as_polars_series(4109954400000)$cast(pl$Datetime("ms", "Europe/Luxembourg"))
#> shape: (1,)
#> Series: '' [datetime[ms, Europe/Luxembourg]]
#> [
#>         2100-03-28 23:00:00 CET
#> ]

>>> import polars as pl
>>> pl.Series("", [4109954400000]).cast(pl.Datetime("ms", "Europe/Luxembourg"))
shape: (1,)
Series: '' [datetime[ms, Europe/Luxembourg]]
[
        2100-03-28 23:00:00 CET
]

[eitsupi]

I think the time zone databases referenced on the R and Rust sides are different. Where each comes from I do not know.
(Trouble is, perhaps this varies by OS.)

[etiennebacher]

For Polars: https://docs.pola.rs/user-guide/transformations/time-series/timezones/

time zone in "area/location" format. See the list of tz database time zones to see what's available.

For R: ?timezones (the whole docs are interesting)

Time zones are a system-specific topic, but these days almost all R platforms use similar underlying code, used by Linux, macOS, Solaris, AIX and FreeBSD, and installed with R on Windows.

Almost all R platforms make use of a time-zone database originally compiled by Arthur David Olson and now managed by IANA, in which the preferred way to refer to a time zone is by a location (typically of a city), e.g., Europe/London, America/Los_Angeles, Pacific/Easter within a ‘time zone region’

[etiennebacher]

I have no clue on how to solve this or even whether this should be reported.

[eitsupi]

Polars does not refer directly to the time zone database, perhaps the chrono crate or something.

What makes the difference is that only on the R side does it change to use CET instead of CEST after 2100, but since we do not know the basis for this, I cannot determine whether the behavior of R or Polars is correct (not to mention that future daylight savings time is not yet determined).