Skip to content

NREL TMY server rejecting python requests #493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wholmgren opened this issue Jun 24, 2018 · 6 comments
Closed

NREL TMY server rejecting python requests #493

wholmgren opened this issue Jun 24, 2018 · 6 comments
Labels
bug
Milestone
0.6.0

Comments

@wholmgren
Copy link
Member

Describe the bug
NREL's TMY file server recently started rejecting requests originating from urlopen. The response was:

"<html><head><title>Request Rejected</title></head><body>An error has occurred. Please consult with your administrator.<br><br>Your support ID is: 12437308730316847769<br><br><a href='javascript:history.back();'>[Go Back]</a></body></html>"

However, it's still possible to download a file from the same url with a web browser. Mostly likely they, intentionally or not, started blocking any requests that don't come with a browser-like HTTP User-Agent header.

I have a fix mostly ready to go.

To Reproduce

pvlib/test/test_tmy.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    @network
    def test_readtmy3_remote():
        url = 'http://rredc.nrel.gov/solar/old_data/nsrdb/1991-2005/data/tmy3/703165TYA.CSV'
>       tmy.readtmy3(url)

pvlib/test/test_tmy.py:19:

        # read in file metadata
        meta = dict(zip(head, csvdata.readline().rstrip('\n').split(",")))

        # convert metadata strings to numeric types
>       meta['altitude'] = float(meta['altitude'])
E       KeyError: 'altitude'

pvlib/tmy.py:178: KeyError

Expected behavior
Should work like it used to!

Versions:
all
@wholmgren wholmgren added the bug label Jun 24, 2018
@wholmgren wholmgren added this to the 0.6.0 milestone Jun 24, 2018
@wholmgren wholmgren mentioned this issue Jun 25, 2018
Merged
6 tasks
cwhanse pushed a commit that referenced this issue Jun 26, 2018
@wholmgren @cwhanse
specify user-agent in tmy3 remote request (#494)
e6aa17c 
Closes #493
@mikofski
Copy link
Member

@wholmgren and @cwhanse unfortunately it looks like NREL has permanently retired the old rredc websites, however I think you can download a single CSV from MTS2:

api_key = 'DEMO_KEY'
blah blah blah

what a PITA

@cwhanse
Copy link
Member

cwhanse commented Jul 26, 2018

Agree with the comment about the NREL site limitations. The PSM NSRDB files have the same restrictions.

At what point does maintenance of this connection to the NSRDB become too much work?

@adriesse
Copy link
Member

Maybe if someone asked, they would keep it alive?

@wholmgren
Copy link
Member Author

At what point does maintenance of this connection to the NSRDB become too much work?

I think the answer is "now". I suggest we remove the feature.

@cwhanse
Copy link
Member

cwhanse commented Jul 26, 2018

I think the answer is "now". I suggest we remove the feature.

I agree - we should remove the capability to pass a url to read_tmy3.

It was nice capability while it worked.

@wholmgren
Copy link
Member Author

NREL's TMY server is back online and the downloads are working again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
0.6.0
Development

No branches or pull requests
4 participants
@mikofski @wholmgren @cwhanse @adriesse