Skip to content

Add map_variables parameter to read_tmy3 function #1623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
May 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
0e60277
Added map_variables param in read_tmy3 func
Dec 21, 2022
a8977fa
solved failing tests
Dec 21, 2022
054dd35
solved failing tests
Dec 22, 2022
3c7f7f6
solving stickerly test fail
Dec 22, 2022
dc6399d
solving stickerly test fail
Dec 23, 2022
773b94e
solving stickerly test fail
Dec 23, 2022
df97faf
added variable_map
Jan 21, 2023
2eee8ce
resolving formatting issues
Jan 21, 2023
7a86c55
Update pvlib/iotools/tmy.py
ooprathamm Jan 22, 2023
7013739
Update pvlib/iotools/tmy.py
ooprathamm Jan 22, 2023
16b3986
Update pvlib/iotools/tmy.py
ooprathamm Jan 22, 2023
5210407
Update pvlib/iotools/tmy.py
ooprathamm Jan 22, 2023
68b54fe
Added tests to map_variables and check depreciation warning
Jan 22, 2023
2ccae9e
resolving formatting issues
Jan 22, 2023
1ee7ff0
resolving formatting issues
Jan 22, 2023
98e53b3
Adding more test cases
Jan 24, 2023
8e5c4d7
Merge branch 'pvlib:main' into main
ooprathamm Jan 24, 2023
96a30a6
updated VARIABLE_MAP
Jan 26, 2023
9a69f12
updates DOIs in tmy.py
Jan 26, 2023
850a798
Merge branch 'pvlib:main' into main
ooprathamm Feb 2, 2023
1e86046
Final changes as requested
Feb 2, 2023
0244732
Fix stickler
AdamRJensen Feb 2, 2023
bccaf02
Update variables_style_rules.csv
AdamRJensen Feb 2, 2023
c4e4417
Add missing map_variables=False
AdamRJensen Feb 2, 2023
b91408a
Add whatsnew deprecation entry
AdamRJensen Feb 2, 2023
94383e9
Apply suggestions from code review
AdamRJensen Feb 7, 2023
2d02f4d
Merge branch 'main' into main
AdamRJensen Feb 7, 2023
c6ebd9e
Add separate if statement for warning
AdamRJensen Feb 7, 2023
616cc76
removed wrong whatsnew entry
Feb 9, 2023
cd9cc86
Update pvlib/tests/iotools/test_tmy.py
ooprathamm Feb 12, 2023
da591f5
Update warning messages according to table
AdamRJensen May 17, 2023
13b8782
Merge remote-tracking branch 'upstream/main' into pr/1623
AdamRJensen May 17, 2023
8f9c788
Update whatsnew
AdamRJensen May 17, 2023
5661f71
Set recolumn=None
AdamRJensen May 17, 2023
a3e2016
Update tests
AdamRJensen May 17, 2023
d0d1644
Update pvlib/iotools/tmy.py
AdamRJensen May 22, 2023
b080c76
Update pvlib/iotools/tmy.py
AdamRJensen May 22, 2023
da131ad
Update docs/sphinx/source/whatsnew/v0.9.6.rst
AdamRJensen May 22, 2023
828b994
Implement updated error message logic from kansersolar
AdamRJensen May 22, 2023
737f5d6
Merge branch 'main' of https://github.com/ooprathamm/pvlib-python int…
AdamRJensen May 22, 2023
1f31390
Clean up deprecation warnings
AdamRJensen May 22, 2023
b720399
Merge branch 'main' into main
AdamRJensen May 22, 2023
98e7891
Fix clearsky.rst
AdamRJensen May 22, 2023
d0c5309
Merge branch 'main' of https://github.com/ooprathamm/pvlib-python int…
AdamRJensen May 22, 2023
24e90d7
Update aod variable mapping
AdamRJensen May 22, 2023
b4bf004
Fix stickler
AdamRJensen May 23, 2023
143f9ac
Update tmy3 variables names in field,description table
AdamRJensen May 23, 2023
5d2ef3a
Update table
AdamRJensen May 24, 2023
fe2f343
More table stuff
AdamRJensen May 24, 2023
35d2d54
Table stuff galore
AdamRJensen May 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/examples/adr-pvarray/plot_simulate_system.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,12 @@
PVLIB_DIR = pvlib.__path__[0]
DATA_FILE = os.path.join(PVLIB_DIR, 'data', '723170TYA.CSV')

tmy, metadata = iotools.read_tmy3(DATA_FILE, coerce_year=1990)
tmy, metadata = iotools.read_tmy3(DATA_FILE, coerce_year=1990,
map_variables=True)

df = pd.DataFrame({'ghi': tmy['GHI'], 'dhi': tmy['DHI'], 'dni': tmy['DNI'],
'temp_air': tmy['DryBulb'], 'wind_speed': tmy['Wspd'],
df = pd.DataFrame({'ghi': tmy['ghi'], 'dhi': tmy['dhi'], 'dni': tmy['dni'],
'temp_air': tmy['temp_air'],
'wind_speed': tmy['wind_speed'],
})

# %%
Expand Down
31 changes: 16 additions & 15 deletions docs/examples/irradiance-decomposition/plot_diffuse_fraction.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@
# of data measured from 1990 to 2010. Therefore we change the timestamps to a
# common year, 1990.
DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
greensboro, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
greensboro, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)

# Many of the diffuse fraction estimation methods require the "true" zenith, so
# we calculate the solar positions for the 1990 at Greensboro, NC.
Expand All @@ -36,8 +37,8 @@
solpos = get_solarposition(
greensboro.index.shift(freq="-30T"), latitude=metadata['latitude'],
longitude=metadata['longitude'], altitude=metadata['altitude'],
pressure=greensboro.Pressure*100, # convert from millibar to Pa
temperature=greensboro.DryBulb)
pressure=greensboro.pressure*100, # convert from millibar to Pa
temperature=greensboro.temp_air)
solpos.index = greensboro.index # reset index to end of the hour

# %%
Expand All @@ -56,10 +57,10 @@
# an exponential relation with airmass.

out_disc = irradiance.disc(
greensboro.GHI, solpos.zenith, greensboro.index, greensboro.Pressure*100)
greensboro.ghi, solpos.zenith, greensboro.index, greensboro.pressure*100)
# use "complete sum" AKA "closure" equations: DHI = GHI - DNI * cos(zenith)
df_disc = irradiance.complete_irradiance(
solar_zenith=solpos.apparent_zenith, ghi=greensboro.GHI, dni=out_disc.dni,
solar_zenith=solpos.apparent_zenith, ghi=greensboro.ghi, dni=out_disc.dni,
dhi=None)
out_disc = out_disc.rename(columns={'dni': 'dni_disc'})
out_disc['dhi_disc'] = df_disc.dhi
Expand All @@ -72,11 +73,11 @@
# developed by Richard Perez and Pierre Ineichen in 1992.

dni_dirint = irradiance.dirint(
greensboro.GHI, solpos.zenith, greensboro.index, greensboro.Pressure*100,
temp_dew=greensboro.DewPoint)
greensboro.ghi, solpos.zenith, greensboro.index, greensboro.pressure*100,
temp_dew=greensboro.temp_dew)
# use "complete sum" AKA "closure" equation: DHI = GHI - DNI * cos(zenith)
df_dirint = irradiance.complete_irradiance(
solar_zenith=solpos.apparent_zenith, ghi=greensboro.GHI, dni=dni_dirint,
solar_zenith=solpos.apparent_zenith, ghi=greensboro.ghi, dni=dni_dirint,
dhi=None)
out_dirint = pd.DataFrame(
{'dni_dirint': dni_dirint, 'dhi_dirint': df_dirint.dhi},
Expand All @@ -91,7 +92,7 @@
# splits kt into 3 regions: linear for kt <= 0.22, a 4th order polynomial
# between 0.22 < kt <= 0.8, and a horizontal line for kt > 0.8.

out_erbs = irradiance.erbs(greensboro.GHI, solpos.zenith, greensboro.index)
out_erbs = irradiance.erbs(greensboro.ghi, solpos.zenith, greensboro.index)
out_erbs = out_erbs.rename(columns={'dni': 'dni_erbs', 'dhi': 'dhi_erbs'})

# %%
Expand All @@ -102,7 +103,7 @@
# exponential correlation that is continuously differentiable and bounded
# between zero and one.

out_boland = irradiance.boland(greensboro.GHI, solpos.zenith, greensboro.index)
out_boland = irradiance.boland(greensboro.ghi, solpos.zenith, greensboro.index)
out_boland = out_boland.rename(
columns={'dni': 'dni_boland', 'dhi': 'dhi_boland'})

Expand All @@ -118,20 +119,20 @@
# file together to make plotting easier.

dni_renames = {
'DNI': 'TMY3', 'dni_disc': 'DISC', 'dni_dirint': 'DIRINT',
'dni': 'TMY3', 'dni_disc': 'DISC', 'dni_dirint': 'DIRINT',
'dni_erbs': 'Erbs', 'dni_boland': 'Boland'}
dni = [
greensboro.DNI, out_disc.dni_disc, out_dirint.dni_dirint,
greensboro.dni, out_disc.dni_disc, out_dirint.dni_dirint,
out_erbs.dni_erbs, out_boland.dni_boland]
dni = pd.concat(dni, axis=1).rename(columns=dni_renames)
dhi_renames = {
'DHI': 'TMY3', 'dhi_disc': 'DISC', 'dhi_dirint': 'DIRINT',
'dhi': 'TMY3', 'dhi_disc': 'DISC', 'dhi_dirint': 'DIRINT',
'dhi_erbs': 'Erbs', 'dhi_boland': 'Boland'}
dhi = [
greensboro.DHI, out_disc.dhi_disc, out_dirint.dhi_dirint,
greensboro.dhi, out_disc.dhi_disc, out_dirint.dhi_dirint,
out_erbs.dhi_erbs, out_boland.dhi_boland]
dhi = pd.concat(dhi, axis=1).rename(columns=dhi_renames)
ghi_kt = pd.concat([greensboro.GHI/1000.0, out_erbs.kt], axis=1)
ghi_kt = pd.concat([greensboro.ghi/1000.0, out_erbs.kt], axis=1)

# %%
# Winter
Expand Down
7 changes: 4 additions & 3 deletions docs/examples/irradiance-transposition/plot_seasonal_tilt.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,13 @@ def get_orientation(self, solar_zenith, solar_azimuth):
# like we expect:

DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
tmy, metadata = iotools.read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
tmy, metadata = iotools.read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)
# shift from TMY3 right-labeled index to left-labeled index:
tmy.index = tmy.index - pd.Timedelta(hours=1)
weather = pd.DataFrame({
'ghi': tmy['GHI'], 'dhi': tmy['DHI'], 'dni': tmy['DNI'],
'temp_air': tmy['DryBulb'], 'wind_speed': tmy['Wspd'],
'ghi': tmy['ghi'], 'dhi': tmy['dhi'], 'dni': tmy['dni'],
'temp_air': tmy['temp_air'], 'wind_speed': tmy['wind_speed'],
})
loc = location.Location.from_tmy(metadata)
solpos = loc.get_solarposition(weather.index)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@
DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'

# get TMY3 dataset
tmy, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
tmy, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)
# TMY3 datasets are right-labeled (AKA "end of interval") which means the last
# interval of Dec 31, 23:00 to Jan 1 00:00 is labeled Jan 1 00:00. When rolling
# up hourly irradiance to monthly insolation, a spurious January value is
Expand Down Expand Up @@ -60,9 +61,9 @@ def calculate_poa(tmy, solar_position, surface_tilt, surface_azimuth):
poa = irradiance.get_total_irradiance(
surface_tilt=surface_tilt,
surface_azimuth=surface_azimuth,
dni=tmy['DNI'],
ghi=tmy['GHI'],
dhi=tmy['DHI'],
dni=tmy['dni'],
ghi=tmy['ghi'],
dhi=tmy['dhi'],
solar_zenith=solar_position['apparent_zenith'],
solar_azimuth=solar_position['azimuth'],
model='isotropic')
Expand Down Expand Up @@ -97,7 +98,7 @@ def calculate_poa(tmy, solar_position, surface_tilt, surface_azimuth):
df_monthly['SAT-0.4'] = poa_irradiance.resample('m').sum()

# calculate the percent difference from GHI
ghi_monthly = tmy['GHI'].resample('m').sum()
ghi_monthly = tmy['ghi'].resample('m').sum()
df_monthly = 100 * (df_monthly.divide(ghi_monthly, axis=0) - 1)

df_monthly.plot()
Expand Down
5 changes: 3 additions & 2 deletions docs/examples/soiling/plot_greensboro_kimber_soiling.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,10 @@
DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'

# get TMY3 data with rain
greensboro, _ = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
greensboro, _ = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
map_variables=True)
# get the rain data
greensboro_rain = greensboro.Lprecipdepth
greensboro_rain = greensboro['Lprecip depth (mm)']
# calculate soiling with no wash dates and cleaning threshold of 25-mm of rain
THRESHOLD = 25.0
soiling_no_wash = kimber(greensboro_rain, cleaning_threshold=THRESHOLD)
Expand Down
11 changes: 6 additions & 5 deletions docs/sphinx/source/user_guide/clearsky.rst
Original file line number Diff line number Diff line change
Expand Up @@ -216,25 +216,26 @@ wavelengths [Bir80]_, and is implemented in

In [1]: tmy_file = os.path.join(pvlib_data, '703165TY.csv') # TMY file

In [1]: tmy_data, tmy_header = read_tmy3(tmy_file, coerce_year=1999) # read TMY data
In [1]: tmy_data, tmy_header = read_tmy3(tmy_file, coerce_year=1999, map_variables=True)

In [1]: tl_historic = clearsky.lookup_linke_turbidity(time=tmy_data.index,
...: latitude=tmy_header['latitude'], longitude=tmy_header['longitude'])

In [1]: solpos = solarposition.get_solarposition(time=tmy_data.index,
...: latitude=tmy_header['latitude'], longitude=tmy_header['longitude'],
...: altitude=tmy_header['altitude'], pressure=tmy_data['Pressure']*mbars,
...: temperature=tmy_data['DryBulb'])
...: altitude=tmy_header['altitude'], pressure=tmy_data['pressure']*mbars,
...: temperature=tmy_data['temp_air'])

In [1]: am_rel = atmosphere.get_relative_airmass(solpos.apparent_zenith)

In [1]: am_abs = atmosphere.get_absolute_airmass(am_rel, tmy_data['Pressure']*mbars)
In [1]: am_abs = atmosphere.get_absolute_airmass(am_rel, tmy_data['pressure']*mbars)

In [1]: airmass = pd.concat([am_rel, am_abs], axis=1).rename(
...: columns={0: 'airmass_relative', 1: 'airmass_absolute'})

In [1]: tl_calculated = atmosphere.kasten96_lt(
...: airmass.airmass_absolute, tmy_data['Pwat'], tmy_data['AOD'])
...: airmass.airmass_absolute, tmy_data['precipitable_water'],
...: tmy_data['AOD (unitless)'])

In [1]: tl = pd.concat([tl_historic, tl_calculated], axis=1).rename(
...: columns={0:'Historic', 1:'Calculated'})
Expand Down
4 changes: 2 additions & 2 deletions docs/sphinx/source/user_guide/timetimezones.rst
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ Let's first examine how pvlib handles time when it imports a TMY3 file.
# some gymnastics to find the example file
pvlib_abspath = os.path.dirname(os.path.abspath(inspect.getfile(pvlib)))
file_abspath = os.path.join(pvlib_abspath, 'data', '703165TY.csv')
tmy3_data, tmy3_metadata = pvlib.iotools.read_tmy3(file_abspath)
tmy3_data, tmy3_metadata = pvlib.iotools.read_tmy3(file_abspath, map_variables=True)

tmy3_metadata

Expand All @@ -287,7 +287,7 @@ print just a few of the rows and columns of the large dataframe.

tmy3_data.index.tz

tmy3_data.loc[tmy3_data.index[0:3], ['GHI', 'DNI', 'AOD']]
tmy3_data.loc[tmy3_data.index[0:3], ['ghi', 'dni', 'AOD (unitless)']]

The :py:func:`~pvlib.iotools.read_tmy2` function also returns a DataFrame
with a localized DatetimeIndex.
Expand Down
8 changes: 8 additions & 0 deletions docs/sphinx/source/whatsnew/v0.9.6.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,16 @@ v0.9.6 (Anticipated June 2023)
Deprecations
~~~~~~~~~~~~

* The ``recolumn`` parameter in :py:func:`pvlib.iotools.read_tmy3`, which maps
TMY3 column names to nonstandard alternatives, is now deprecated.
We encourage using ``map_variables`` (which produces standard pvlib names) instead.
(:issue:`1517`, :pull:`1623`)

Enhancements
~~~~~~~~~~~~
* Added ``map_variables`` argument to the :py:func:`pvlib.iotools.read_tmy3` in
order to offer the option of mapping column names to standard pvlib names.
(:issue:`1517`, :pull:`1623`)
* Update the URL used in the :py:func:`pvlib.iotools.get_cams` function. The new URL supports load-balancing
and redirects to the fastest server. (:issue:`1688`, :pull:`1740`)
* :py:func:`pvlib.iotools.get_psm3` now has a ``url`` parameter to give the user
Expand Down Expand Up @@ -49,5 +56,6 @@ Contributors
* Siddharth Kaul (:ghuser:`k10blogger`)
* Kshitiz Gupta (:ghuser:`kshitiz305`)
* Stefan de Lange (:ghuser:`langestefan`)
* :ghuser:`ooprathamm`
* Kevin Anderson (:ghuser:`kandersolar`)

3 changes: 3 additions & 0 deletions pvlib/data/variables_style_rules.csv
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ dni_extra;direct normal irradiance at top of atmosphere (extraterrestrial)
dhi;diffuse horizontal irradiance
bhi;beam/direct horizontal irradiance
ghi;global horizontal irradiance
ghi_extra;horizontal irradiance at top of atmosphere (extraterrestrial)
gri;ground-reflected irradiance
aoi;angle of incidence between :math:`90\deg` and :math:`90\deg`
aoi_projection;cos(aoi)
Expand All @@ -32,6 +33,8 @@ relative_humidity;relative humidity
wind_speed;wind speed
wind_direction;wind direction
pressure;atmospheric pressure
albedo;ratio of reflected solar irradiance to global horizontal irradiance, unitless
precipitable_water;total precipitable water contained in a column of unit cross section from earth to top of atmosphere
v_mp, i_mp, p_mp;module voltage, current, power at the maximum power point
v_oc;open circuit module voltage
i_sc;short circuit module current
Expand Down
Loading