removed gsutil dependency

[loretoparisi]

I have removed the gsutil dependency using curl and google drive. This approach is well-known and used in several frameworks that need to download large models files (like in FastText)

[FloridaPete]

Thanks for doing this, but can you answer a dumb question? How do you download the model with this technique? I see access denied when trying to follow the url's. Thanks.

[loretoparisi]

@FloridaPete the URL is public, basically curl will get the url from the public download url of the google drive file. Did you run the shell script?

[Georgerowberry]

I'm getting this error when running the script:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

[loretoparisi]

@Georgerowberry thank you, let me check the tar file I have uploaded.

[WuTheFWasThat]

Hi - this is nice, but I'm afraid I can't have us officially pointing to something not controlled by us (in Google Drive)

[loretoparisi]

Of course you should upload your file, but if you prefer to force people to install 200MB of google-cloud-kit, I think that it's up to you. Normally when dealing with large dataset, most of github repository uses Google Drive approach by the way, if you are not aware of this.

The right question is: why OpenAI has anything better to host files?!?

[ZeweiChu]

I'm getting this error when running the script:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

I had the same error with your script.

I ended up downloading from the google drive directly.
https://drive.google.com/uc?export=download&id=1kY-qc0uCU3uGPhVGTfryvHr5zWh_B9g7

[loretoparisi]

@ZeweiChu @Georgerowberry I have fixed the link, that it points to the right 117M.tar.gz file!
By the way the script is here https://github.com/loretoparisi/gpt-2/blob/master/download_model.sh

[WuTheFWasThat]

Sorry I don't have time to test right now but can a version work simply with something like

curl -X GET https://storage.googleapis.com/gpt-2/models/117M/encoder.json?