Requirement for `configure_s3_access` is inconsistent between `rio` and `default` drivers

[alexgleith]

If you load data using the default driver, you must use configure_s3_access from the datacube libary, i.e.,:

from datacube.utils.aws import configure_s3_access

configure_s3_access(requester_pays=True)
dc = Datacube() 

dc.load(
    ...
)
...

Whereas if you use the rio driver, then it needs to be imported from odc.stac.configure_s3_access.

from odc.stac import configure_s3_access

configure_s3_access(requester_pays=True)
dc = Datacube() 

dc.load(
    ...
    driver="rio"
)
...

and this is rather unintuitive!

[robbibt]

Could this function could be moved to odc-loader so the same tool could be accessible from both datacube-core and odc-stac?

I'd also love to know more about when to use configure_s3_access vs. from odc.stac import configure_rio(which I think also sets cloud defaults) - e.g. do we need to use both?

[alexgleith]

do we need to use both

I think you only need to use configure_s3_access, but yeah, it's different depending on the loader/driver used.

[SpacemanPaul]

odc.stac is already a requirement. We can just use that.

Edit: In fact, odc-stac is already importing it from odc-loader (of course)

[SpacemanPaul]

I'd also love to know more about when to use configure_s3_access vs. from odc.stac import configure_rio(which I think also sets cloud defaults) - e.g. do we need to use both?

The code in core is a mess with code in both utils.rio and utils.aws importing dynamically from the other.

I'll see how much I can clean it up by porting the whole tangled mess to the odc-loading implementation without breaking anything and keeping at least token backwards compatibility.

And to your explicit question - if you call configure_s3_access with cloud defaults, you do NOT need to call configure_rio as well.

[SpacemanPaul]

Actually I think we're stuck with this until we retire the legacy loader or at least until after the next release. The way that the legacy loader and odc-loader handle RIO config in a distributed environment is too different.

For this release I can make the internal datacube.utils.aws.configure_s3_access take an optional driver parameter that routes to the odc-loader implementation for the "rio" driver.