Skip to content

NREL bird clearsky #276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 35 commits into from
May 26, 2017
Merged

NREL bird clearsky #276

merged 35 commits into from
May 26, 2017

Conversation

mikofski
Copy link
Member

@mikofski mikofski commented Dec 3, 2016

Yet another clear sky model, based on the MS Excel workbook by NREL Daryl Myers and the model described in the paper "A Simplified Clear Sky model for Direct and Diffuse
Insolation on Horizontal Surfaces"
by R.E. Bird and R.L Hulstrom, SERI Technical
Report SERI/TR-642-761, Feb 1981. Solar Energy Research Institute, Golden, CO.

bird_clearsky_model

* move clearsky.py into clearsky/core.py just for fun, let's try it out
* fix path to data in clearsky to LinkeTurbidities.mat
* add BIRD workbook to data for testing

Signed-off-by: Mark Mikofski <[email protected]>
Copy link
Member

@wholmgren wholmgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I think the Bird model would be a useful addition, though I admit I don't have any experience with it.

I'll put the API-level comments here. The resolution of these might make some of my line-by-line comments irrelevant.

  1. Restructuring clearsky.py into a package: maybe we should do this first in a separate PR? On a related note, I thought you were an proponent of api.py in modules are too long and have too many symbols #235 so I was a little surprised that you didn't use that here. It's not clear to me why bird would be in its own module while the other algorithms remain in the same module.
  2. I'm not sure about the nrel.py module name. Do people frequently associate this algorithm with NREL?
  3. The bird function does a lot: calculates solar position, airmass, atmospheric properties, ET irradiance, and finally clear sky. I think the function and its signature should be simplified by requiring users to do more work before they get here. This would also be consistent with the way the existing clear sky algorithms work. So, I think the signature should be more like

bird(zenith, airmass_relative, aod700=0.1, precipitable_water=1., ozone=0.3, pressure=101325., dni_extra=1364.)

This is also related to the ineichen refactoring that I did in #199. The documentation can make note of the methods that need to be used on the input data in order to exactly reproduce the traditional model.

Should we strip out the test data from the excel file and put it in a csv file? Advantages are that the csv file is smaller and we avoid the xlrd dependency. More importantly, are we allowed to redistribute the NREL file?

Why add the requirements file? setup.py already has the minimum pvlib requirements in it.

def bird(doy, hr, lat, lon, tz, press_mB, o3_cm, h2o_cm, aod_500nm, aod_380nm,
b_a, alb, lyear=365.0, solar_constant=1367.0):
"""
Bird Simple Clearsky Model
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use numpydoc

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️

np.sin(dec_rad) * np.sin(lat_rad)
)
zenith = np.rad2deg(ze_rad)
airmass = np.where(zenith < 89,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same as relativeairmass(method='kasten1966')

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • removed this calc in bird since passing relative airmass as argument
  • use this in test_clearsky.test_bird()
    ✔️

return pd.DataFrame({'Eb': Eb, 'Ebh': Ebh, 'Gh': Gh, 'Dh': Dh}, index=dt)


if __name__ == "__main__":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, we don't have any script code in pvlib, but I would consider keeping this in a test module.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed script - it was just for debugging
✔️

return solar_constant * (a0 + a1 + a2 + a3 + a4)


def test_bird():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests should go in a module named test_[__name__].py. We should probably make a tests subsubpackage within each subpackage e.g. pvlib/clearsky/tests/test_bird.py

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️

return id_, id_nh, gh, gh - id_nh, testvalues


def etr(doy, lyear=365.0, solar_constant=1367.0):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This the same as irradiance.extraradiation(method='spencer')

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using dni_extra arg now, so this is removed, but I'll use it in the test
✔️

)
pstar = press_mB / patm
am_press = pstar*airmass
t_rayliegh = np.where(airmass > 0,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are any of the np.where statements repetitive? In any case, calculate the airmass > 0 mask once and reuse it in subsequent tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not necessary anymore, since relative airmass is NaN if not valid
✔️

)
gh = np.where(airmass > 0, (id_nh + ias) / (1.0 - alb * rs), 0.0)
testvalues = (day_angle, declination, eqt, hour_angle, zenith, airmass)
return id_, id_nh, gh, gh - id_nh, testvalues
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to do the subtraction on a separate line.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, changed to diffuse_horiz
✔️

Solar Radiation Model
From the publication "A Simplified Clear Sky model for Direct and Diffuse
Insolation on Horizontal Surfaces" by R.E. Bird and R.L Hulstrom, SERI Technical
Report SERI/TR-642-761, Feb 1991. Solar Energy Research Institute, Golden, CO.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1981

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️

Insolation on Horizontal Surfaces" by R.E. Bird and R.L Hulstrom, SERI Technical
Report SERI/TR-642-761, Feb 1991. Solar Energy Research Institute, Golden, CO.

The model is based on comparisons with results from rigourous radiative transfer
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

text needs to be edited so that it's not specific to the excel file. Not sure if there is a licensing/attribution issue here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️


def bird(doy, hr, lat, lon, tz, press_mB, o3_cm, h2o_cm, aod_500nm, aod_380nm,
b_a, alb, lyear=365.0, solar_constant=1367.0):
"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some (all?) of the edited doc string above should be moved here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️

@jforbess
Copy link
Contributor

jforbess commented Dec 5, 2016 via email

* compare eot from spencer and pvcdrom to spa-python
* add note about coefficients in spencer 1971 and more references
* clean up docstrings for bird model, fix pub date to 1981, add references section, add attribution to daryl myers
* add spaces between colon and parameter to ineichen and simplified_solis models to correspond to numpydoc style
* revert changes in 457615d, change pvlib-path back to dirname of clearsky.py file, remove added clearsky_path
* start to change bird function signature to conform to existing clear sky algos, and strip out some of the complicated algos like the relative airmass, declination and solpos
 * change args aod_380nm to aod380 and aod_500nm to aod500.
 * clean up eot docstrings, add references, use external linking
* added both cooper-69 and spencer-71
* replacing calls in bird to calls to new formulas for declination, eot, dayangle
* remove day_angle, relative airmass, eot, zenith, declination, extraterrestrial dni calcs, and pass values in as needed
* change o3_cm to ozone, h2o_cm to precipitable_water, b_a to asymmetry and alb to albedo
* replace pstar, patm and am_press with absolute airmass method from atmosphere
* testvalues no longer needed
* use irradiance.extraradiation() and remove etr()
* add analytical solar zenith angle and hour angle methods to solar position module
* also add returns
* remove defaults for aod, since should be supplied
TODO: change input to broadband when linke turbidity PR is complete
* remove times, not needed anymore
* remove commented relative airmass(method="kasten1966") since passed as arg
* remove extraradiation(method="spencer") since passed as arg
* change etr_ to etr since no more etr
TODO: change all variables to easier names
* add seinfeld and pandis to declination references
* finish hour angle calc
* also add returns
* remove defaults for aod, since should be supplied
TODO: change input to broadband when linke turbidity PR is complete
* remove times, not needed anymore
* remove commented relative airmass(method="kasten1966") since passed as arg
* remove extraradiation(method="spencer") since passed as arg
* change etr_ to etr since no more etr
TODO: change all variables to easier names
* add seinfeld and pandis to declination references
* finish hour angle calc
* save ze_cos once, instead of recalculating
* simplify/shorten some lines now that np.where(airmass > 0,<calc>,0.0) is gone
* add to what's new for v0.4.4
* add new solar position formulas and bird clear sky to api
* move links to references, and space references to make numpydoc happy
* add see also for eot, declination and solar_zenith_analytical
@mikofski
Copy link
Member Author

mikofski commented Jan 14, 2017

I think this is done. Some notes:

  • I reversed the API changes. I'll leave that to you. Personally I like to use the dunder classes for breaking up modules into packages, but the api.py idea seems interesting too.

  • I refactored the bird model to remove any redundant solarposition and atmosphere stuff, like you said. For testing I added some basic solar position stuff like analytical zenith, equation of time and declination

  • I also refactored the signature to match other clearsky models

  • if you really want to remove the xls bird file in lieu of a csv, that's fine, but it is a primary source, and I don't think there are any license issues, it is publicly available without registering, but guess we should check on that. I would think like 90% of users will already have xlrd, PITA for build testing huh

  • the requirements file is really really useful!

    1. if you weren't using conda, travis would just look at your requirements and build your venv from that
    2. it makes it really easy to make a venv, which is what I've done for pvlib and why I made the file. Most projects recommend it, EG see this
  • either python 3.4 or 3.5 randomly fails for no reason, something in test_forecast using netCDF4, but it doesn't always happen, and sometimes it changes from one version to another. What's up?

@wholmgren
Copy link
Member

Sorry again for the delay @mikofski. Does anyone have time to look at this more carefully?

Without looking closely (and thus running a high chance of saying something stupid)...

I reversed the API changes.

API looks good, thanks. I like the idea of reorganizing with api.py or something else, but let's do it in a separate PR.

I refactored the bird model to remove any redundant solarposition and atmosphere stuff, like you said. For testing I added some basic solar position stuff like analytical zenith, equation of time and declination

I was initially unhappy to see all of this less precise solar position functionality in a clear sky model PR. But I guess I'm ok with these functions if we document that users should prefer other functions for high precision calculations. Maybe put them in a different subsection in the API documentation?

if you really want to remove the xls bird file in lieu of a csv, that's fine, but it is a primary source, and I don't think there are any license issues, it is publicly available without registering, but guess we should check on that. I would think like 90% of users will already have xlrd, PITA for build testing huh

I like plain text for smaller data because people can view it on github, in diffs, and their editors. But this is fine.

the requirements file is really really useful!

The base requirements are numpy 1.9, pandas 0.14, pytz, and six. Maybe you want to make two requirements files -- requirements and requirements-all? Why the newer numpy scipy version limits?

either python 3.4 or 3.5 randomly fails for no reason, something in test_forecast using netCDF4, but it doesn't always happen, and sometimes it changes from one version to another. What's up?

see #293.

Are the keyword parameters sufficiently tested? Does it work with scalars?

- use csv file instead of xls file - NOTE: I made this file by
changing all cells' format to General, fitting column size to content
width to show all significant figures, then saving as csv, new file
is less than 370kb vs 1.2mb
- use pandas read_csv instead of xlrd, way easier :)
- remove requirements.txt, can't be bothered
@mikofski
Copy link
Member Author

still working on it. Any comments from anyone?

Maybe put them in a different subsection in the API documentation

good idea, I'll put them in a section after spa with text prefixing that says these are low accuracy solar position methods

I like plain text for smaller data because people can view it on github, in diffs, and their editors

remove xls, and reversed xlrd dependencies everywhere and use csv instead

The base requirements are numpy 1.9, pandas 0.14, pytz, and six. Maybe you want to make two requirements files -- requirements and requirements-all? Why the newer numpy scipy version limits?

I removed the requirements.txt, too much trouble if no one cares but me, the version dependencies were accidental, but there were other dependencies to build sphinx for example, and ephem, etc, it's just not worth it

Are the keyword parameters sufficiently tested? Does it work with scalars?

Nope, still need to check on this

* add Daryl R. Myers, "Solar Radiation" CRC Press (2013) reference
* fix some sphinx-build errors in solar-position, explicit emphasis
start with no end for "*Note" (missing closing "*" for emphasis in
`nrel_earthsun_distance` and `spa_python`
* use numpydoc linked references for spencer71 equation of time
* add BIRD_08_16_2012_patm.csv for 2nd bird test data
* fix docs by moving text before parameters so not indented
@mikofski
Copy link
Member Author

@wholmgren and others:

This is ready for your review now:

  1. moved low precision solar position correlations to separate section of documentation on API.rst
  2. added keyword and scalar tests - passing

@wholmgren
Copy link
Member

Thanks, Mark. Might be next week before I can look carefully. Other users should feel free to chime in.

Copy link
Member

@wholmgren wholmgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for long delay in getting to this @mikofski. I think we're close.

It would be great if you could add a new section to the clearsky.rst file and update the other sections appropriately, but I worry that might be asking too much for this PR. We could leave it as a stub in the documentation and create a separate issue to remind us to get to it eventually.

solarposition.solar_zenith_analytical
solarposition.declination_spencer71
solarposition.declination_cooper69
solarposition.equation_of_time_Spencer71
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lower case s here and in function definition.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☑️

Parameters
----------
zenith : numeric
Solar zenith angle in degrees.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solar_zenith not apparent - in case I forget

NREL Bird Simple Clear Sky Broadband Solar Radiation Model

Based on original implementation by Daryl R. Myers at NREL.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could add some notes about how zenith and airmass should be calculated if users want to exactly reproduce the entire bird model.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or this could go into a new section in the clearsky.rst documentation.


Returns
-------
direct_beam : numeric
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more conventional in pvlib to return a DataFrame or OrderedDict, rather than a tuple. Series in, DataFrame out, anything else in, OrderedDict out. The other functions in this module demonstrate this pattern. columns/keys should follow http://pvlib-python.readthedocs.io/en/latest/variables_style_rules.html

@wholmgren wholmgren added this to the 0.4.5 milestone Feb 15, 2017
@mikofski
Copy link
Member Author

@wholmgren sorry for being so slow. I am still working on this.

@mikofski
Copy link
Member Author

@wholmgren and @cwhanse I think this is finally ready to be merged. It passes all tests. I've addressed the return types by converting them to OrderdDict or DataFrame and ordered the output similar to other clear sky models. I did not address the concerns about adding documentation to clearsky.rst about the inputs to duplicate the Daryl Myers Excel spreadsheet, or add any new significant documentation other that to the API. Perhaps I can address additional documentation as needed in a future realease? Thanks!

@mikofski
Copy link
Member Author

@wholmgren I made minor documentation changes to address the testing parameters and solar or apparent zenith argument. Please let me know if there are any other issues to address before you can merge this. I would like to use the Bird model for comparisons of AOD/Pwat data with Linke turbidity in with Ineichen's original and newer simplified-Solis clearsky model for our upcoming PVSC paper. 😉

@wholmgren
Copy link
Member

wholmgren commented May 18, 2017 via email

Copy link
Member

@wholmgren wholmgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How big (MB) are these test files?

@@ -6,6 +6,7 @@ v0.4.4 (February 18, 2017)
Enhancements
~~~~~~~~~~~~

* Added NREL Bird clear sky model. (:issue:`276`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're onto 0.4.5

airmass = airmass_relative
# Bird clear sky model
am_press = atmosphere.absoluteairmass(airmass, pressure)
t_rayliegh = (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rayleigh not rayliegh

@@ -367,3 +367,15 @@ def test_nrel_earthsun_distance():
expected = pd.Series(np.array([0.983289204601]),
index=pd.DatetimeIndex([times, ]))
assert_series_equal(expected, result)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we add tests for all of the other functions you've added to solarposition.py? I think they should be marked private if you don't want to add the tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added 2 more tests for low accuracy declination and solar-zenith formulas.

@mikofski
Copy link
Member Author

@wholmgren the test files are each 360-kb, so total combined less than 1-mb, okay?

screenshot from 2017-05-24 20-32-45

* move whats new note from 0.4.4 to 0.4.5
* test declination and solar-zenith
* correct spelling of Rayleigh
@wholmgren
Copy link
Member

ok on the test file size.

I think this is ready to go. Agreed?

@mikofski
Copy link
Member Author

Unless anyone else has any comments, yes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants