Skip to content

Support resumable/chunked uploads #26

@heralden

Description

@heralden

We considered supporting uploading via the CLI intermine_boot client, but after some discussion found that it would be more user-friendly and probably just as reliable to use chunked uploads in the browser.

One good library that provides this is flow.js.
We will need to have the upload endpoint support the API.

The next question would be where to save the files.

  • Upload to intermine_compose and save file to local filesystem (how it's done currently)
  • Upload to intermine_compose and use GCP’s app engine cloud client library to save to cloud bucket (we should probably move over to this, as the files will be large and numerous; the previous approach should still be used when this is not available due to running locally with intermine_boot)
  • Upload directly to Google Cloud Storage bucket (a different alternative; might be more difficult and not sure if it's worth the extra effort)
    • This can be done for large files using either flow.js (issue, PR) or GCS directly via resumable uploads API (info, guide)

Note that the files need to be available to both the wizard and InterMine instance without being duplicated. (With a GCP bucket: after acquiring an OAuth token, files can be downloaded from buckets with a GET request.) This could be a problem with our current Auth approach, so we may need to explore using Daniela’s OAuth server

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions