add custom user agent for download_url #3498
Conversation
torchvision/datasets/utils.py
Outdated
| def _urlretrieve(url: str, filename: str, **kwargs: Any) -> None: | ||
| request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) | ||
| with open(filename, "wb") as fh, urllib.request.urlopen(request) as response: | ||
| fh.write(response.read()) |
There was a problem hiding this comment.
urllib.request.urlretrieve does not provide an option to set custom headers.
| import urllib | ||
|
|
There was a problem hiding this comment.
urllib is part of the standard library so there is no reason to import it lazily.
Codecov Report
@@ Coverage Diff @@
## master #3498 +/- ##
==========================================
- Coverage 78.71% 78.65% -0.06%
==========================================
Files 105 105
Lines 9711 9723 +12
Branches 1555 1557 +2
==========================================
+ Hits 7644 7648 +4
- Misses 1576 1583 +7
- Partials 491 492 +1
Continue to review full report at Codecov.
|
| pbar.update(chunk_size) | ||
| fh.write(chunk) | ||
|
|
||
|
|
||
| def gen_bar_updater() -> Callable[[int, int, int], None]: |
There was a problem hiding this comment.
We can probably remove / deprecate this now since we are not using it anymore.
| - name: Install PyTorch from the nightlies | ||
| run: | | ||
| pip install numpy | ||
| pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html | ||
| - name: Install torch nightly build | ||
| run: pip install --pre torch -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html | ||
|
|
||
| - name: Install torchvision | ||
| run: pip install -e . |
There was a problem hiding this comment.
This is a quick and dirty fix and should be revisited in a follow-up PR.
Summary: * add custom user agent for download_url * fix progress bar * lint * [test] use repo instead of nightly for download tests Reviewed By: NicolasHug, cpuhrsch Differential Revision: D26945735 fbshipit-source-id: 6030e0927dda95afb63f32aceacfc3b479bf3f9f
Fixes #3497.