NCDC FTP Site

[nickrsan]

Name: NOAA Full NCDC Site
Organization: NOAA NCDC
Description URL:
Download URL: https://www1.ncdc.noaa.gov/pub/data/
File Types:
Size:
Status: In progress - mirroring by non-GitHub user
NCEI pub data mirrored by Azimuth Project: https://bitbucket.org/azimuth-backup/azimuth-inventory/issues/40/noaa-ncei-complete-pub-directory

[bkirkbri]

nClimGrid subset is issue #116

ftp://ftp.ncdc.noaa.gov/pub/data/climgrid/

[bkirkbri]

Local Climatological Data (LCD) subset is issue #117

ftp://ftp.ncdc.noaa.gov/pub/data/lcd/

[bkirkbri]

Paleoclimatology subset is issue #17

ftp://ftp.ncdc.noaa.gov/pub/data/paleo/

We received a report that the https://www1.ncdc.noaa.gov/pub/ is 12 Tb. We grabbed 620 Gb, but don't know the source, one Nick Gregory, and why he would know the size. Can anyone vouch for that number?

I just did a full directory walk of that ncdc.noaa.gov/pub, and got 29.620 Tb, 1325435 files, and 11686 folders. Thanks for any help people intended.

We cannot do all of this. Is there some sense to dividing it up? Please advise to climate -at- mm -dot- st. Thanks!

[bkirkbri]

I posted some subsets above. I agree it's best to break up what's left. I can claim some of them. Do you have sizes for top-level directories?

Thanks!

I can get these tomorrow. Tracking as Azimuth Backup Kickstarter Project Issue #77.

I am awaiting the /pub/data total but here, in the interim, is what I have. It's been running since mid-afternoon.

Note this is probably a lower bound. I received a number of 500 error codes during the run of the du against these directories, and, so, there were files whose sizes were missed.
I will update when I have the final. The number above for /pub/data was another 30 Tb, but we'll see.

[mejackreed]

I can potentially grab some. What is left?

The FTP site remains in a "being copied" state. That said, it is not clear exactly where we are. We do have 3.9 Tb of it.