-
Notifications
You must be signed in to change notification settings - Fork 1.1k
ENH: geometric calculations for sunrise, sunset, and transit #583
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENH: geometric calculations for sunrise, sunset, and transit #583
Conversation
… and sun transit times
pvlib/solarposition.py
Outdated
|
||
def _hours(times, hour_angle, longitude, equation_of_time): | ||
timezone = times.tz.utcoffset(times).total_seconds() / 3600. | ||
return (hour_angle - longitude - equation_of_time/4.)/15. + 12. + timezone |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E226 missing whitespace around arithmetic operator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from pep8: whitespace in expressions:
If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies). Use your own judgment; however, never use more than one space, and always have the same amount of whitespace on both sides of a binary operator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fine with me here and below.
pvlib/test/test_solarposition.py
Outdated
sunrise = (sr.time() for sr in sst[0]) | ||
sunset = (ss.time() for ss in sst[1]) | ||
transit = (tr.time() for tr in sst[2]) | ||
sunrise_hours = [sr.hour + (sr.minute + sr.second/60.)/60. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E226 missing whitespace around arithmetic operator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from pep8: whitespace in expressions:
If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies). Use your own judgment; however, never use more than one space, and always have the same amount of whitespace on both sides of a binary operator.
pvlib/test/test_solarposition.py
Outdated
transit = (tr.time() for tr in sst[2]) | ||
sunrise_hours = [sr.hour + (sr.minute + sr.second/60.)/60. | ||
for sr in sunrise] | ||
sunset_hours = [ss.hour + (ss.minute + ss.second/60.)/60. for ss in sunset] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E226 missing whitespace around arithmetic operator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from pep8: whitespace in expressions:
If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies). Use your own judgment; however, never use more than one space, and always have the same amount of whitespace on both sides of a binary operator.
pvlib/test/test_solarposition.py
Outdated
sunrise_hours = [sr.hour + (sr.minute + sr.second/60.)/60. | ||
for sr in sunrise] | ||
sunset_hours = [ss.hour + (ss.minute + ss.second/60.)/60. for ss in sunset] | ||
transit_hours = [tr.hour + (tr.minute + tr.second/60.)/60. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E226 missing whitespace around arithmetic operator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from pep8: whitespace in expressions:
If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies). Use your own judgment; however, never use more than one space, and always have the same amount of whitespace on both sides of a binary operator.
@thunderfish24 ask an ye shall receive |
note: test failure for py3.6 is unrelated |
By the way, the same formula is in Duffie and Beckman and in probably many other solar textbooks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks mark!
pvlib/solarposition.py
Outdated
@@ -1210,3 +1210,50 @@ def hour_angle(times, longitude, equation_of_time): | |||
)).total_seconds() / 3600. for t in times]) | |||
timezone = times.tz.utcoffset(times).total_seconds() / 3600. | |||
return 15. * (hours - 12. - timezone) + longitude + equation_of_time / 4. | |||
|
|||
|
|||
def _hours(times, hour_angle, longitude, equation_of_time): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like a more descriptive function name or a one line documentation for _hours
and _times
.
pvlib/solarposition.py
Outdated
|
||
def _hours(times, hour_angle, longitude, equation_of_time): | ||
timezone = times.tz.utcoffset(times).total_seconds() / 3600. | ||
return (hour_angle - longitude - equation_of_time/4.)/15. + 12. + timezone |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fine with me here and below.
pvlib/solarposition.py
Outdated
def _times(times, hours): | ||
return np.array([(dt.timedelta(hours=h) + t.tz.localize( | ||
dt.datetime(t.year, t.month, t.day) | ||
)) for h, t in zip(hours, times)]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This loop using pure python is likely to be painfully slow and defeat the purpose of an analytical approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I can use numpy arrays with timedelta64[s]
here instead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice catch @wholmgren , I get about a 70x speed improvement (from 43.5 ms to 605us) to by using numpy datetime64
and timedelta64[ns]
. I wonder where else we could use these? There is an oddity that hours
is a Float64Index
from pandas, not just floats, which bothers me, but ok for now.
See this script that shows improvements with numpy over looping.
In [1]: import pandas as pd
In [2]: import datetime as dt
In [3]: import numpy as np
In [4]: d = pd.DatetimeIndex(start='2013-01-01 00:00:00', end='2013-12-31 23:59:59', freq='H', tz='Etc/GMT+8')
In [5]: h = np.loadtxt('../sunrise_sunset_transit.txt', dtype=np.dtype([('sunrise', float), ('sunset', float), ('transit', float)]))
In [6]: sr = pd.Float64Index(h['sunrise'])
In [7]: %paste
def _times(times, hours):
return np.array([(dt.timedelta(hours=h) + t.tz.localize(
dt.datetime(t.year, t.month, t.day)
)) for h, t in zip(hours, times)])
## -- End pasted text --
In [8]: %timeit _times(d, sr)
43.5 ms ± 354 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [9]: del _times
In [10]: %paste
def _times(times, hours):
"""helper converts hours from an array of floats to localized times"""
tz_info = times.tz
return pd.DatetimeIndex(
times.tz_localize(None).values.astype('datetime64[D]')
+ (hours.values * 3600. * 1.e9).astype('timedelta64[ns]'), tz=tz_info)
## -- End pasted text --
In [11]: %timeit _times(d, sr)
605 µs ± 5.07 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
pvlib/test/test_solarposition.py
Outdated
@@ -15,6 +16,9 @@ | |||
from conftest import (requires_ephem, needs_pandas_0_17, | |||
requires_spa_c, requires_numba) | |||
|
|||
DIRNAME = os.path.dirname(__file__) | |||
PROJDIR = os.path.dirname(DIRNAME) | |||
DATADIR = os.path.join(PROJDIR, 'data') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably better to put these in the function if they're not going to be reused elsewhere.
pvlib/test/test_solarposition.py
Outdated
transit_hours = [tr.hour + (tr.minute + tr.second/60.)/60. | ||
for tr in transit] | ||
test_data_file = os.path.join( | ||
DATADIR, 'sunrise_sunset_transit.txt') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need such a large data file to comprehensively test the algorithm? I am guessing 10-20 well chosen points is sufficient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess not, but I thought others might want to use this, it's a full year of hourly sunrise/sunset/transit times from NREL SPA. It's about 600kb, is that too much? Whatever you think is best, maybe ditch it completely, and just test 5-10 values hardcoded in the test?
pvlib/solarposition.py
Outdated
sunrise = _times(times, sunrise_hour) | ||
sunset = _times(times, sunset_hour) | ||
transit = _times(times, transit_hour) | ||
return (sunrise, sunset, transit) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should try to make the API consistent with the existing function. So this should be changed to support dict or DataFrame output or we can discuss changing the API of the other function elsewhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking that this was a lower level helper function, and that a higher level api function like get_sunrise
would call it and based on the how
argument, and then package up the return values into a DataFrame as needed. But happy to change this to output a DataFrame for now if that's what the consensus is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm in favor of returning a pandas object rather than the tuple, and also, the intermediate function get_next_sunrise
with a how
argument that wraps the geometric, pyephem and spa methods. The input is a tz-aware datetime and location. get_sunrise
is somewhat ambiguous: assume the datetime is solar noon at the location. Is the desired output the sunrise of the current day, or the next sunrise (the following day)? get_sunrise(date)
clarifies things but requires the function to localize date (for pyephem) or determine the corresponding date in UTC time (for spa). For these reasons I'm suggesting the next
approach.
To complete the function set we'd add get_previous_sunrise
, get_next_sunset
, etc.
pvlib/solarposition.py
Outdated
|
||
|
||
def _hours(times, hour_angle, longitude, equation_of_time): | ||
timezone = times.tz.utcoffset(times).total_seconds() / 3600. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm odd to see times
as both the object and explicit argument of its own method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is weird, but I think it's the only way, I broke it up into two lines that explains a bit more:
tz_info = times.tz # this is the pytz timezone info object that has the utcoffset method
tz = tz_info.utcoffset(times) # this method requires a datetime-like argument to operate on
pvlib/solarposition.py
Outdated
|
||
Parameters | ||
---------- | ||
times : :class:`pandas.DatetimeIndex` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please remove the rst markup here. it detracts from readability when inspecting the function signature outside of the rendered docs.
pvlib/solarposition.py
Outdated
def sunrise_sunset_transit_analytical(times, latitude, longitude, declination, | ||
equation_of_time): | ||
""" | ||
Analytical calculation of solar sunrise, sunset, and transit. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't care for the term analytical
but I don't have a better one to suggest. Perhaps the references mentioned have a better name for this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may want to clean up the naming convention in this module. We have prefixes 'spa' and 'nrel' for functions using SPA, we sometimes have 'pyephem' but not always, and use the suffix 'analytical' (I'm Ok with this) for some purely geometric methods.
I prefer _, e.g., declination_cooper69
. where there are multiple options to calculate quantity.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"geometric" might be an alternative to "analytical"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really like "geometric" but would we change zenith and azimuth to match?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd say yes, to follow the naming pattern, tacitly labeling the collection of spherical trigonometry methods as the 'geometric' model.
Deprecate _analytical
to 0.7, I'd say.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I like it! Everyone else in agreement? It'll have to wait until after SPI, but I'll try to get back to this asap. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like "geometric". We should use "geometric" for the new function in this PR, but let's take on the deprecation of the existing functions in a separate PR.
* use hardcoded values for 10-20 expected values * remove DIRNAME, etc * make stickler happy * no rst markup in numpydoc parameters * add references to duffie/beckman and Frank Vignola * get pytz timezone info first so it's not weird * add returns to docstring
pvlib/test/test_solarposition.py
Outdated
@@ -1,5 +1,6 @@ | |||
import calendar | |||
import datetime | |||
import os |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
F401 'os' imported but unused
* add one-liners to explain helpers * remove import os from test
pvlib/solarposition.py
Outdated
tz_info = times.tz | ||
return pd.DatetimeIndex( | ||
times.tz_localize(None).values.astype('datetime64[D]') | ||
+ (hours.values * 3600. * 1.e9).astype('timedelta64[ns]'), tz=tz_info) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
W503 line break before binary operator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pep8: Should a line break before or after a binary operator?
To solve this readability problem, mathematicians and their publishers follow the opposite convention. Donald Knuth explains the traditional rule in his Computers and Typesetting series:
“Although formulas within a paragraph always break after binary operations and relations, displayed formulas always break before binary operations”
In Python code, it is permissible to break before or after a binary operator, as long as the convention is consistent locally. For new code Knuth's style is suggested.
emphasis mine
Notes:
do new features have to support the minimum requirements? I can see that patching old code should be backwards compatible, but if you are installing pvlib now to use an analytical sunrise/sunset equation, then you will have the newest pandas |
And FYI: pandas-0.14.0 isn't even compatible with numpy-1.10.1 according to conda
|
* if old pandas change tz directly, and hours is already a numpy array * if new pandas then use tz_localize to strip tz, and change hours from Float64Index to numpy array * move operator to end to match existing style in pvlib
@wholmgren I am seeing some backwards compatibility issues on Travis with numpy and pandas minimum requirements
Since there are so many API changes in v0.6 can we consider maybe updating some of the minimum requirements now? Pandas is currently v0.23.4 and NumPy is at 1.15, keeping really old requirements seems limiting to me a little. Thanks. |
Also, can we maybe consider dropping support for Python-3.4? |
I am all in favor of raising the minimums if there is a reason to do so. But I take these incompatibilities as sign that the approach might be too complicated and will be difficult to maintain in the future. Let's think about if this is really the implementation that we want. Somewhere you commented something to the effect of the spec'ed minimum numpy/pandas versions are incompatible -- is it that they are incompatible or that Continuum doesn't supply pre-built binaries for them?
We can consider this too but not in this pull request. |
My guess is that much of the logic can be simplified if we do some combination of the following
|
it seems to be working now, only had to make 2 minor changes to satisfy the CI
I was wrong, you were right, according to
I am already assuming that the times must be tz-aware and that they have a tz or tz_info field
good idea, like Linux time, can we pursue this as a course of last resort? The calculations seem to be working now, I believe that internally that is what
Do you mean otherwise, (minus what's new) I think this is ready for review |
BTW: pandas-0.14.0 is from 2014, IMO we should not support this because it's not a v1.0 release, so a version that is 4 years old may have more issues than just incompatibility.
did you see this comment, the simpler approach was 70x slower, I don't want to create a maintenance headache, but I don't think this approach is too complicated. |
pvlib/solarposition.py
Outdated
Analytical calculation of solar sunrise, sunset, and transit. | ||
|
||
.. warning:: The analytic form neglects the effect of atmospheric | ||
refraction. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably "geometric" instead of analytic. Also guessing it neglects more than refraction, but haven't looked at the sources.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also spherical earth, sun is represented by a point at its center, ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here and in the other "analytical"/"geometrical" functions should we give an estimate of the typical magnitude of the errors? Something like "The error depends on location and time of year but is of order 10 minutes."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from the old test the max differences are
- sunrise 0.11 hours
- sunset 0.11 hours
- transit 0.02 hours
this is a rough estimate, only to 2 decimals, I manually iterated the 2nd decimal to see what tolerance I could get.
Also the errors are one sided, ie: for sunrise the geometric estimate is always earlier up to 0.11 hours, and down to some positive, non-zero number, so there's always some sunrise error that is a minute or so earlier than the apparent sunrise, and ie: for sunset it's always later up to 0.11 hours, and down to a positive, non-zero number, so there's always some sunset error that is a minute or so later than the apparent sunset.
pvlib/solarposition.py
Outdated
tz_info = times.tz | ||
try: | ||
hours = hours.values # make a new reference | ||
except AttributeError: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand the point of this try/except. Is hours
not always the same type? If not, suggest we change _hours()
to do the right thing with something like
hours = # your eqn
# enforce Series
hours = pd.Series(hours, index=times)
# OR enforce array
hours = np.array(hours)
return hours
Then we can remove this try except.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no apparently depending on the pandas version it changes, sometimes it's a numpy array of floats and sometimes it's a pandas series of pandas.Float64Index
, I didn't bother to track down which versions are which, but it seems to be in the 0.teen versions, I think >0.20 yields series. I'm sure I could enforce the type somewhere else, as an output of some calculation, but I didn't bother to
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would simplify things to enforce it elsewhere. I am sorry for being slow/stubborn but these functions are hard for me to follow. I assume it will be even harder for future me to understand them when something inevitably breaks.
pvlib/solarposition.py
Outdated
try: | ||
# OLD pandas, "TypeError: Already tz-aware, use tz_convert to convert." | ||
times.tz = None | ||
except AttributeError: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also don't understand this try/except. Your API declaration requires a localized DatetimeIndex, so times.tz_convert
is guaranteed to work. This is true for pandas 0.14 and 0.23.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tz_convert
changes the timezone from one to another, ie: it applies the tz_utcoffset
so eg: from Etc/GMT+8
to UTC
would add 8 hours to all of the times.
But that's not what we want. We just want to make the tz-aware time a local naive time, so we want to effectively replace tz_info
with None
.
Again in some older version of pandas, you could just do this by setting tz=None
, and in newer ones you can call tz_localize(None)
to convert a tz-aware time to a naive one, without applying a utc offset.
EG: 2017-01-01T12:30:00-0800
just becomes 2017-01-01T12:30:00
same time, but no more timezone
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand now, thanks.
newer ones you can call tz_localize(None) to convert a tz-aware time to a naive one, without applying a utc offset.
Is this a documented behavior of tz_localize
? I can't find official support for it. If not I recommend a different implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe it is documented, I've used this before, I don't recall where, but I will look for it.
@cwhanse and @wholmgren thanks so much for being patient and providing really great feedback, most of which I believe I have addressed and incorporated.
- enforce hours as numpy array in
_hours()
instead of in_times()
- break up calculation into separate steps so they are more clear
- use "geometric" instead of "analytical"
- in warning, explain circular orbit with sun as center point, and state max errors of 6 minutes or so
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wholmgren it's in the pandas documentation for DatetimeIndex.tz_localize
Passing
None
will remove the time zone information preserving local time.
It's and also here:
Passing
None
will remove the time zone information preserving local time.
and there
None will remove timezone holding local time.
@cwhanse normalize
seems very interesting, and perhaps what we're both looking for? Since v0.16, see what's new.
Convert times to midnight.
The time component of the date-timeise converted to midnight i.e. 00:00:00. This is useful in cases, when the time does not matter. Length is unaltered. The timezones are unaffected.
* change name in solarposition.py and test_solarposition.py * enforce that hours is a numpy array in _hours instead of in _times * add more useful comments, separate into steps to explain process
thanks for feedback!, tests are passing! all good now? |
where is this conversation in github??? these line by line conversations
are great until they're horrible.
I guess I was looking at the neither/nor docs. Normalize seems useful, but
does it do something different than .floor('1D')? I'm ok bumping the pandas
version to support this.
added: of course the email response goes straight to the main thread because even Github can't figure it out.
…On Mon, Sep 24, 2018 at 2:35 PM Mark Mikofski ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In pvlib/solarposition.py
<#583 (comment)>:
> + tz_info = times.tz # pytz timezone info
+ tz = tz_info.utcoffset(times).total_seconds() / 3600.
+ return (hour_angle - longitude - equation_of_time / 4.) / 15. + 12. + tz
+
+
+def _times(times, hours):
+ """helper converts hours from an array of floats to localized times"""
+ tz_info = times.tz
+ try:
+ hours = hours.values # make a new reference
+ except AttributeError:
+ pass
+ try:
+ # OLD pandas, "TypeError: Already tz-aware, use tz_convert to convert."
+ times.tz = None
+ except AttributeError:
@wholmgren <https://github.com/wholmgren> it's in the pandas
documentation for DatetimeIndex.tz_localize
<https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DatetimeIndex.tz_localize.html#pandas.DatetimeIndex.tz_localize>
Passing None will remove the time zone information preserving local time.
It's and also here
<https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.dt.tz_localize.html#pandas.Series.dt.tz_localize>
:
Passing None will remove the time zone information preserving local time.
and there
<https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Timestamp.tz_localize.html>
None will remove timezone holding local time.
but neither here
<https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.tz_localize.html#pandas.Series.tz_localize>
nor there
<https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.tz_localize.html#pandas.DataFrame.tz_localize>
@cwhanse <https://github.com/cwhanse> normalize
<https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.dt.normalize.html#pandas.Series.dt.normalize>
seems very interesting, and perhaps what we're both looking for? Since
v0.16 <pandas-dev/pandas#10047>, see what's new
<https://pandas.pydata.org/pandas-docs/version/0.16/whatsnew.html>.
Convert times to midnight.
The time component of the date-timeise converted to midnight i.e.
00:00:00. This is useful in cases, when the time does not matter. Length is
unaltered. The timezones are unaffected.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#583 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AELiRy-lTAjOrpBEJyT4lVYRf52x0JTLks5ueVALgaJpZM4WxIcW>
.
|
If I understand correctly, pandas 0.16 adds normalize for Timestamps and Series. 0.14 does have it for DatetimeIndex: >>> times = pd.DatetimeIndex(start='20180924 1600', end='20180924 1700', freq='1H', tz='America/Phoenix')
>>> times
<class 'pandas.tseries.index.DatetimeIndex'>
[2018-09-24 16:00:00-07:00, 2018-09-24 17:00:00-07:00]
Length: 2, Freq: H, Timezone: America/Phoenix
>>> times.normalize()
<class 'pandas.tseries.index.DatetimeIndex'>
[2018-09-24 00:00:00-07:00, 2018-09-24 00:00:00-07:00]
Length: 2, Freq: None, Timezone: America/Phoenix
>>> pd.__version__
'0.14.0' |
* also remove utcoffset(), unreliable with pandas dataframes, only really works with scalar datetimes and timestamps * instead use tz_localise(None) to get naive, but still local time, and subtract the localized time, to get a sequence of timezones
pvlib/solarposition.py
Outdated
"""converts hour angles in degrees to hours as a numpy array""" | ||
naive_times = times.tz_localize(None) # naive but still localized | ||
tzs = 1 / (3600. * 1.e9) * ( | ||
naive_times.astype(np.int64) - times.astype(np.int64)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E126 continuation line over-indented for hanging indent
pvlib/test/test_solarposition.py
Outdated
@@ -718,3 +718,56 @@ def test_hour_angle(): | |||
# sunrise: 4 seconds, sunset: 48 seconds, transit: 0.2 seconds | |||
# but the differences may be due to other SPA input parameters | |||
assert np.allclose(hours, expected) | |||
@pytest.fixture() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E302 expected 2 blank lines, found 0
* remove geometric expected sunrise/sunset/transite test fixture * check abs tolerance between geometric and spa are less than max absolute error difference which is typically 4-6 minutes for sunrise/sunset, and about 10-40 seconds for transite * move _times_to_hours to solarposition, and use the same approach as other hour/time conversions, use pandas>= tz_localize(None) or normalize to get the local, naive times, and time at previous midnight, then convert to np.int64, and then convert to hours from nanoseconds * since fixture is a dataframe, get column, convert to MST timezone, before calling _times_to_hours(<pd.DatetimeIndex>) * get times from expected_rise_set_spa.index * get lat/lon from golden_mst test_fixture
pvlib/test/test_solarposition.py
Outdated
@@ -531,7 +531,9 @@ def test_get_solarposition_altitude(altitude, expected, golden): | |||
"delta_t, method, expected", [ | |||
(None, 'nrel_numpy', expected_solpos_multi()), | |||
(67.0, 'nrel_numpy', expected_solpos_multi()), | |||
pytest.mark.xfail(raises=ValueError, reason = 'spa.calculate_deltat not implemented for numba yet') | |||
pytest.mark.xfail( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E122 continuation line missing indentation or outdented
pvlib/test/test_solarposition.py
Outdated
pytest.mark.xfail(raises=ValueError, reason = 'spa.calculate_deltat not implemented for numba yet') | ||
pytest.mark.xfail( | ||
raises=ValueError, | ||
reason = 'spa.calculate_deltat not implemented for numba yet') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E251 unexpected spaces around keyword / parameter equals
@wholmgren , @cwhanse , ready to merge I think,
|
question: should we define
somewhere, to save time? it's calculated in a few places now |
Thanks @mikofski. Ok with me if you want to define I'd like to see this PR tested against the pandas 0.15 changes in #595. Probably easiest to wait until that goes in. I want to test that one myself, probably on Wednesday. Sorry for the backlog. midnite or midnight? Seems like we should prefer the standard spelling. I'm less confident working with the Can you revise the name for this PR? Also a few items on the checklist to be completed. |
* wrap long lines, link to issues * add constant NS_PER_HR and substiture for (3600. * 1.e9) * change arg name to hourangle avoid shadowing function hour_angle() * fix spelling in _local_times_from_hours_since_midnight() * use normalize() and astype(np.int64) instead of naive_times.astype('datetime64[D]').astype('datetim64[ns]') and midnight_times + (hours*NS_PER_HR).astype('timedelta64[ns]') for stability and consistency with other hour/times conversion functions
pvlib/solarposition.py
Outdated
# sunrise, sunset, and transit | ||
return pd.DatetimeIndex( | ||
(naive_times.normalize().astype(np.int64) | ||
+ (hours * NS_PER_HR).astype(np.int64)).astype('datetime64[ns]'), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
W503 line break before binary operator
@wholmgren updated api and what's new docs, and removed All tests pass in both environments with pandas>=0.15.0 & numpy>=1.10.1: py27-minvirtualenv
tests
pip freeze
py3xtests
conda list
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great, thanks for putting in the extra effort for the manual pandas 0.15 testing. In that case I think it's ready to go. I'd like to give @cwhanse a chance to review it again given his detailed comments.
pvlib/test/test_solarposition.py
Outdated
@@ -718,3 +721,45 @@ def test_hour_angle(): | |||
# sunrise: 4 seconds, sunset: 48 seconds, transit: 0.2 seconds | |||
# but the differences may be due to other SPA input parameters | |||
assert np.allclose(hours, expected) | |||
|
|||
|
|||
def test_geometric_sunrise_sunset_transit(expected_rise_set_spa, golden_mst): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change function name to test_sunrise_sunset_transit_geometric
pvlib/test/test_solarposition.py
Outdated
|
||
|
||
def test_geometric_sunrise_sunset_transit(expected_rise_set_spa, golden_mst): | ||
"""Test analytical calculations for sunrise, sunset, and transit times""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
geometric
rather than analytical
pvlib/solarposition.py
Outdated
tz=tz_info) | ||
|
||
|
||
def _times_to_hours(times): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the function name should be more specific, perhaps _hours_after_local_midnight
The function description should also be more specific.
pvlib/solarposition.py
Outdated
|
||
|
||
def _local_times_from_hours_since_midnight(times, hours): | ||
"""converts hours from an array of floats to localized times""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This implies (I think) that hours
is hours since local (?) midnight? I think the function description should define what is expected for times
and hours
. times
only serves to communicate a date. hours
appears to be hours since midnight.
pvlib/solarposition.py
Outdated
Parameters | ||
---------- | ||
times : pandas.DatetimeIndex | ||
Corresponding timestamps, must be timezone aware. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The description of times
needs to specify localized to the timezone for latitude
and longitude
, it would help to state that sunrise, sunset and transit are calculated for the date part of times
.
* Created :py:func:`pvlib.iotools.read_srml` and | ||
:py:func:`pvlib.iotools.read_srml_month_from_solardat` to read University of | ||
Oregon Solar Radiation Monitoring Laboratory data. (:issue:`589`) | ||
* Add geometric functions for sunrise, sunset, and sun transit times, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wasn't going to pick this nit but since I have other comments, let's sort this list to group the sunrise etc. changes together and the iotools
items together.
* _times_to_hours -> _times_to_hours_after_local_midnight * change corresponding usage in test_solarposition.py and wrap some long lines * in docstring specify that times should be localized to the timzeone of the lat/lon args in hour_angle() and sunrise_sunset_transit_geometric() * change arg in solar_azimuth_analytical() and solar_zenith_analytical() from hour_angle -> hourangle to avoid shadowing func name hour_angle() from outer scope and remove redundant parentheses from return
* change description to say "geometric" vs. analytical * improve description of _local_times_from_hours_since_midnight to be more specific by adding "hours since midnight"
@cwhanse thanks for your feedback, I believe I've addressed your comments. |
FYI: the appveyor error is unrelated, caused by http error
|
@wholmgren OK to merge? |
Thanks Mark! |
… and sun transit times
pvlib python pull request guidelines
Thank you for your contribution to pvlib python! You may delete all of these instructions except for the list below.
You may submit a pull request with your code at any stage of completion.
The following items must be addressed before the code can be merged. Please don't hesitate to ask for help if you're unsure of how to accomplish any of the items below:
docs/sphinx/source/api.rst
for API changes.docs/sphinx/source/whatsnew
file for all changes.Brief description of the problem and proposed solution (if not already fully described in the issue linked to above):