Skip to content

Switch to an arrow-like partition structure for bundled data #5

@Bisaloo

Description

@Bisaloo

Data is partitioned across 3 axes and making this structure explicit by using an arrow-like partition structure may be helpful:

inst/extdata/
├── 2017
│   ├── all
│   ├── home
│   ├── other
│   ├── school
│   └── work
└── 2020
    ├── all
    │   ├── all
    │   ├── rural
    │   └── urban
    ├── home
    │   ├── all
    │   ├── rural
    │   └── urban
    ├── other
    │   ├── all
    │   ├── rural
    │   └── urban
    ├── school
    │   ├── all
    │   ├── rural
    │   └── urban
    └── work
        ├── all
        ├── rural
        └── urban

If we made this switch, we may as well store the data as parquet and use arrow for better performance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions