Use xarray.open_dataset() for password-protected Opendap files

[jenfly]

I've been using xarray.open_dataset() to read Opendap netcdf files from NASA's MERRA-2 data archive. Recently they changed their site so that now you must enter a username and password to read any files. They describe here how to access data with Pydap: http://disc.sci.gsfc.nasa.gov/registration/registration-for-data-access#python.

I experimented with a similar approach (adding username and password to the url) with xarray.open_dataset() and specifying engine='pydap', but no luck. Is there a way to use xarray.open_dataset() to read password-protected Opendap files? Thanks!

[shoyer]

If you write engine='pydap' in open_dataset, the URL should be passed directly on to pydap, but you'll still need to follow all of their other instructions. If you're getting an error message from xarray, let us know but otherwise I'm at a loss -- you should check with the folks at NASA.

[jenfly]

Thanks very much for your reply! I still get an error from xarray when I use the engine='pydap' option. Here's a minimum (almost) working example (almost because you need an account with the server so you can substitute your username/password into the url string):

import xarray
from pydap.client import open_url

url = 'http://<username>:<password>@goldsmr5.sci.gsfc.nasa.gov/opendap/MERRA2/M2I3NPASM.5.12.4/1986/01/MERRA2_100.inst3_3d_asm_Np.19860101.nc4'

ds1 = open_url(url)    # Works but data isn't in xarray format
ds2 = xarray.open_dataset(url, engine='pydap')    # Error message, see attached

I've attached the error message here --
error_msg.txt
I don't know enough about the inner workings of xarray to trace through it. Please let me know if any of this means anything to you and has a reasonably easy fix or workaround. Thank you!

[shoyer]

If the dataset has a "time" dimension, try accessing the first few values. Can you view them in pydap? Xarray's open_dataset does a little more work than pydap's open_url, insofar as it actually downloads some array data.

[jenfly]

Ah, I see. Thanks for the suggestion. Using Pydap I'm able to see all the variables and their metadata, so I thought it was working, but when I try to actually access the data values, I get the same error message as from Xarray. The issue must be something unrelated to Xarray -- I'll keep investigating. Thanks for your help!

[j08lue]

@jenfly did you find a solution how to make opendap authentication work with xarray? Might be worthwhile posting it here, even though the issue has to do with the backends.

[jenfly]

@j08lue no, not yet. I've been in touch with the folks at NASA who run the server, but their suggestions didn't work for me and I haven't had time to keep troubleshooting. I will need to sort out this issue in the next couple of months to get some data that I need, so if/when I ever resolve it, I'll post the solution here.

[jenfly]

I've finally found something useful online and am able to use Pydap to open these files -- hoping someone can help me find a way to integrate this into an xarray.open_dataset() function call and then I will be a very happy camper!

Turns out much of the info posted by NASA online is out of date and based on a different implementation of Pydap than what is actually being used currently (argh). Here is something that actually works, from http://www.pydap.org/en/latest/client.html#urs-nasa-earthdata:

from pydap.client import open_url
from pydap.cas.urs import setup_session

url = 'https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2/M2T1NXSLV.5.12.4/2016/06/MERRA2_400.tavg1_2d_slv_Nx.20160601.nc4'

session = setup_session(username, password)
dataset = open_url(url, session=session)

where I've assigned the username and password variables with the appropriate values in another function.

I've tested this and it is working, but I would prefer to do things within Xarray since all my code is already using it. Just for fun, I tried ds = xarray.open_dataset(url, engine='pydap', session=session), to see if the extra keyword would be magically sent to the pydap engine, but got an error message. Is there a way to incorporate this functionality into xarray.open_dataset? Thank you so much for any assistance!

[rabernat]

Hi @jenfly, it's great to see that you have tracked down this root issue! I agree we should be able to support direct access to these sort of opendap resources within xarray. It should not be too tricky to implement, and in fact, if you are interested, it could be a great opportunity for you to open a pull request and become directly involved in the project. We would be very happy to gain another contributor.

You can see the line where pydap.open_url gets called here:
https://github.com/pydata/xarray/blob/master/xarray/backends/pydap_.py#L64

We just need a mechanism to pass the username and password from open_dataset to the pydap backend. There are two possible options I see:

1. we could add new username and password keyword args to open_dataset. This is the most straightforward, but open_dataset already has a ton of arguments, so maybe it is not ideal.
2. we could parse out the username and password from a url like https://username:password@... within the pydap backend.

It would be good to get some other opinions on which approach would be preferable.

[jenfly]

Thanks, @rabernat! I'd be happy to try implementing this in the project. I'm a newbie when it comes to contributing to big projects like this (so far I've just used Github for my own little projects) so I might have some naive questions as I figure out how things work.

The two options you mentioned for passing username and password info to open_dataset both sound good to me. I don't have any strong preference between them. How do I get other opinions on which approach to use? Should I start a new issue thread?

Also, I realized that there is another hiccup along the way. When I try to specify engine='pydap' in open_dataset, I get the same error message as mentioned in #1174, that the object has no attribute iteritems. When I wrote the first post in this thread, back in October, I was able to use engine='pydap' without any problems. This seems to be related to recent upstream changes in Pydap: pydap/pydap#43 and I presume might require more substantial changes either in Xarray or Pydap so that they can work together again. Any thoughts on how to handle this?

[shoyer]

Parsing username/password from the URL would be very easy to add.

We need to figure out a solution for the proliferating arguments on open_dataset before we add many more, so I would prefer that for now.

Another option is to add session as an argument on xarray.backends.PydapDataStore, and encourage passing PydapDataStore objects into xarray.open_dataset for extra customizability, e.g.,

store = xarray.backends.PydapDataStore(url, session)
ds = xarray.open_dataset(store)

[shoyer]

Pydap has a new v3.2 release, but it still needs some fixes to work with xarray -- or xarray needs to be updated to work with the new version of pydap. I think pydap/pydap#48 once merged would probably be enough to restore xarray compatibility.

[laliberte]

I like the idea of passing PydapDataStore objects that include the session object. It seems more likely to be forward compatible, especially if Central Authentication Services multiply (as one would expect) with different authentication mechanisms.

[jenfly]

I also like the idea of passing PydapDataStore objects that include the session object. Delving deeper into the pydap authentication, I found that there are already several different setup_session functions available to create the session object, corresponding to different authentication procedures (pydap.cas.get_cookies.setup_session, pydap.cas.urs.setup_session, pydap.cas.esgf.setup_session) as well as additional arguments to setup_session beyond username and password. Best to deal with all this separately with pydap rather than trying to embed it within xarray.

I'm still having problems trying to get xarray.open_dataset to work with pydap. Using the latest commit on pydap/master (in which pydap/pydap#48 is merged) I'm now getting a new error: AttributeError: '<class 'pydap.model.BaseType'>' object has no attribute 'encode'. When I have some time, I'll look into it further and try to see what else is needed to restore compatibility.