Embargoed dandisets assets are not showing up in "Assets Summary"

[weiglszonja]

I have an issue with two embargoed dandisets (here and here), both have multiple assets uploaded but the "Assets Summary" is empty.

I tried to use dandi service-scripts reextract-metadata https://dandiarchive.org/dandiset/001264/draft to update the asset metadata, but it still shows an empty "Assets Summary".

Is there another way to force updating the metadata for the dandisets?

[waxlamp]

Hi @weiglszonja, thank you for the bug report. This does seem to be a bug--we'll get on this as soon as we can.

[weiglszonja]

Thank you @waxlamp, let me know if you need any further info.

[pauladkisson]

If it helps, my dandiset also has this issue: https://dandiarchive.org/dandiset/001259/draft

[dandibot]

🚀 Issue was released in v0.4.5 🚀

[mvandenburgh]

Hi @weiglszonja and @pauladkisson the problem was that some assets in your dandisets are failing validation, but DANDI was not properly displaying validation errors for embargoed dandisets. We've released a fix for this, so you should now see those on your dandisets. Please let us know if you have any further questions about these validation errors or anything else!

[pauladkisson]

@mvandenburgh, thank you for looking into this. I'm seeing validation errors now on the dandiset page, but I still can't stream the nwbfiles for my example notebooks (see dandi/example-notebooks#106). Furthermore, the error that I'm getting is Value error, An embargo end date is required for NIH awards to be in compliance with NIH resource sharing policy. which seems like it should be a dandiset metadata error rather than an asset error...

For my own understanding, this dandiset is in progress (still in draft), does it really need an embargo end date before it's even published?

[waxlamp]

mvandenburgh, thank you for looking into this. I'm seeing validation errors now on the dandiset page, but I still can't stream the nwbfiles for my example notebooks (see dandi/example-notebooks#106).

I believe that may be a different error related to accessing embargoed data without user credentials. Since the data is embargoed, the server will only enable access to the file in S3 if you make the request as your logged in user (otherwise, you will get a 403 error as you seem to be getting).

Unfortunately I don't know enough about how the notebook access to data works, so perhaps I have this wrong. But on the surface it does seem to be a different problem. (@kabilar may be able to help more directly.)

Furthermore, the error that I'm getting is Value error, An embargo end date is required for NIH awards to be in compliance with NIH resource sharing policy. which seems like it should be a dandiset metadata error rather than an asset error...

We are working on solving this problem. The trouble is that as a user, there is nothing in the system you can do to resolve this; we're looking for a reasonable path that will sort of self-resolve so that you no longer see that error message.

For my own understanding, this dandiset is in progress (still in draft), does it really need an embargo end date before it's even published?

The short answer is "yes" in order to comply with, e.g., NIH embargo rules.

Currently, only unpublished (i.e., draft-only) Dandisets can be embargoed, since one of the conditions of ending the embargo period is publication of the data (generally, this would mean publishing results, but for DANDI it also means creating a published version of the Dandiset). Aside from ending embargo early via publication, NIH has a 12-month limit on embargo, which is where the need to set that date comes from.

Sorry for the trouble and confusion here. Hopefully we'll have a resolution soon.