specify user-agent in tmy3 remote request #494
Conversation
|
@wholmgren Thanks for tracking this one down. |
|
@cwhanse any concerns here? if not, please go ahead and "squash and merge". |
| response = urlopen(filename) | ||
| if filename.startswith('http'): | ||
| request = Request(filename, headers={'User-Agent': | ||
| 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) ' |
There was a problem hiding this comment.
Does any user agent work here, or does it need to match the user's browser and platform? If the latter then we need to test this fix on multiple OS.
There was a problem hiding this comment.
No and no.
By default, Python's urllib specifies a user-agent that is "Python-urllib/x.y" where x.y is the version. https://docs.python.org/3/howto/urllib2.html This user agent is no longer allowed. I am guessing most user agents that identify themselves as scripts are now blocked.
The automated tests run on linux and windows (not mac), and they pass on this PR (note that they currently fail on the master). I don't think that the server has any way of knowing that our script is pretending to be a browser unless it does something smarter (e.g. sees that the requests are coming from IPs associated with travis-ci and appveyor).
git diff upstream/master -u -- "*.py" | flake8 --diffand/or landscape.io linting service.docs/sphinx/source/api.rstfor API changes.docs/sphinx/source/whatsnewfile for all changes.