PERF: Speed up DatetimeIndex field accessors with ZoneInfo timezones#64377
Open
shubhamgoel27 wants to merge 4 commits intopandas-dev:mainfrom
Open
PERF: Speed up DatetimeIndex field accessors with ZoneInfo timezones#64377shubhamgoel27 wants to merge 4 commits intopandas-dev:mainfrom
shubhamgoel27 wants to merge 4 commits intopandas-dev:mainfrom
Conversation
jbrockmendel
reviewed
Mar 3, 2026
Route ZoneInfo timezones through the fast C-level binary search path for UTC-to-local conversion instead of the slow per-timestamp Python tzinfo API path. This is done by: 1. Fixing tz_cache_key() to return tz.key for ZoneInfo, enabling DST transition data caching. 2. Adding a ZoneInfo branch in get_dst_info() that reuses dateutil's transition data via dateutil_gettz(tz.key), avoiding the need to re-implement TZif parsing. 3. Updating Localizer.__cinit__ to route ZoneInfo through the DST binary search path (use_dst=True) instead of the slow tzlocal path (use_tzlocal=True). This provides ~100x speedup for DatetimeIndex.month, .year, .day, .hour and other field accessors when using ZoneInfo timezones. Closes pandas-dev#64363 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
c2a0729 to
34d8a0e
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The tz_cache_key function was returning tz.key for ZoneInfo objects,
which could collide with pytz's tz.zone for the same IANA key (e.g.
both return "US/Pacific"). This caused get_dst_info to return
dateutil-derived transition data for pytz timezones when ZoneInfo was
cached first, leading to incorrect LMT offset handling.
Prefix ZoneInfo cache keys with "zoneinfo/" to match the namespacing
pattern used by dateutil ("dateutil" + filename).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bb17170 to
b91c65a
Compare
Member
|
I suspect this would also close #58962 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tz.utcoffset()API pathdateutil_gettz(tz.key)— no custom TZif parsing neededtz_cache_key()to returntz.keyfor ZoneInfo, enabling DST transition data cachingRoot cause
In
Localizer.__cinit__(tzconversion.pyx), ZoneInfo was grouped withtzlocal:This forced every timestamp through
_tz_localize_using_tzinfo_api(), which makes a Python-leveltz.utcoffset()call per element — O(n) Python API calls instead of a single O(n) C-level loop with O(log k) binary search on cached transition arrays.Fix
tz_cache_key(): Returntz.keyfor ZoneInfo (was returningNone, preventing caching)get_dst_info(): Add ZoneInfo branch that converts to the equivalent dateutil timezone viadateutil_gettz(tz.key)and reuses dateutil's transition data extraction (dateutil reads the same underlying TZif files)Localizer.__cinit__: Removeis_zoneinfo(tz)fromuse_tzlocalcheck so ZoneInfo flows throughget_dst_info→ binary search pathBenchmark
Correctness verified against both dateutil and pytz across multiple timezones including DST spring-forward and fall-back transitions.
Closes #64363
Test plan
.year,.month,.day,.houraccessors🤖 Generated with Claude Code