Add import and export of OAISTree

[pdurbin]

As I mentioned at IQSS/dataverse#5235 (comment) I'm curious if the "DVTree" (Dataverse Tree) format could be used to upload sample data to a brand new Dataverse installation for use in demos and usability testing.

I would love to see some docs. Or a pointer to the code for now. Thanks! 😄

[skasberger]

The structure so far is not defined, it's just a rough idea I had, inspired by @petermr CTree structure. I definitely want to talk with some of the Dataverse Devs about the idea -> if this would work right now and in the long run. The idea in general is, that only the filenames and the structure of the folders and files tell about, what should/can be inside and how to treat different files then. Like, every dataverse folder must have a metadata file, same for datasets. The content of the metadata file then must not be strictly defined, but most likely will also have mandatory attributes. This then can be used to create a local export independent of OS and connecting programming language, which also can be used by humans.

Here my first draft of the structure:

Naming Conventions:

• Dataverse: dv_IDENTIFIER, prefix dv_, id = alias
• Dataset: ds_IDENTIFIER, prefix ds_, id = id
• Datafile: FILENAME

├── dv_harvard/
│     ├── metadata.json
│     └── dv_iqss/
│            ├── metadata.json
│            └── ds_microcensus-2018/
│                    ├── metadata.json
│                    └── datafiles/
│                           ├── documentation.pdf
│                           └── data.csv
│     └── metadata.json
└── dv_aussda/
       └── ds_survey-labour-2016/
              ├── metadata.json
              └── datafiles/
                     ├── docs.pdf
                     └── data.tsv

Some open questions:

• are the filenaming conventions compatible, possibly? e.g. is it always okay/possible to convert the dataverse alias to a filename string and store it on every operating system?
• is the filename the best identifier for the datafiles? or is it's hash better?
• how to handle versioning? is a DVTree only for one version possible or should there be another level of folders, like v1/?
• do we need to seperate metadata into 1) general metadata and 2) metadata for API upload (add api.json or so)?

[pdurbin]

@skasberger thanks for this great write up! I just posted on the (ancient) "Round tripping the contents of DVN" thread at https://groups.google.com/d/msg/dataverse-community/07h0Ca-Ai1I/qyq3l-lakc0J with a link to this issue. I'm hoping to spur some good discussion. 😄

[skasberger]

As mentioned several times at the Dataverse Community Conference: BagIt seems to be very similar, and could be a good inspiration. https://en.wikipedia.org/wiki/BagIt

[skasberger]

After developing the first proof of concept, I recommend to rename it to OAISTree (Open Archival Information System), cause the related processes are the guidance for the directory structure and it's conventions. Here my actual draft.

ROOT_DIR
├── YYYYMMDD_dataverses.csv
├── YYYYMMDD_datasets.csv
├── YYYYMMDD_datafiles.csv
├── terms-of-access.html
├── terms-of-use.html`
├── PICKE_FILE.pickle: Pickle Files
└── OAISTrees/
        └──DATASET_ID/
               ├── DATASET_ID_history.json
               └── SIP/
                       └── RAW_DATA_FILENAME
               └── AIP/
                       ├── DATASET_ID_metadata.json
                       └── DATASET_ID_DATAFILE_ID_metadata.json
               └── DIP/
                       ├── terms-of-use.html
                       └── terms-of-access.html
        └──DATASET_ID/
        └──DATASET_ID/
        └──DATASET_ID/

[petermr]

Is this dependent on the https://en.wikipedia.org/wiki/Open_Archival_Information_System ? This is a standard and a community.

…

On Mon, Dec 2, 2019 at 2:05 PM Stefan Kasberger ***@***.***> wrote: After developing the first proof of concept, I recommend to rename it to OAISTree (Open Archival Information System), cause the related processes are the guidance for the directory structure and it's conventions. Here my actual draft. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/AUSSDA/pyDataverse/issues/5?email_source=notifications&email_token=AAFTCSZSAMQTEYBUPFIUXALQWUI2ZA5CNFSM4HNLTYGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFTS4PI#issuecomment-560410173>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFTCSZS6TQNAXMITURUUXLQWUI2ZANCNFSM4HNLTYGA> .

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

[skasberger]

Hi Peter,

not really dependent. It's a first proposal for a standardized folder/data structure for OAIS related data, which then can be used to convert data from and to different systems. We use Dataverse, but others use iRODS or other software solutions.

[petermr]

Fine, so Dataverse encapulates OAIS. The important thing in using standards is to be consistent with their specs otherwise it accuses confusion with software.

…

On Mon, Dec 2, 2019 at 2:20 PM Stefan Kasberger ***@***.***> wrote: Hi Peter, not really dependent. It's a first proposal for a standardized folder/data structure for OAIS related data, which then can be used to convert data from and to different systems. We use Dataverse, but others use iRODS or other software solutions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/AUSSDA/pyDataverse/issues/5?email_source=notifications&email_token=AAFTCS4O7KHWREZUJYIMZRDQWUKRTA5CNFSM4HNLTYGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFTUI7A#issuecomment-560415868>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFTCS5TKOHHTMIYVCBVKILQWUKRTANCNFSM4HNLTYGA> .

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK