Add tools for working with filers #62
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds a bunch of tools for working with filers directly using S3 (with botocore available in base environment) and using the mc command more easily. Documentation for these tools available on internal GitLab, can be copied to this pull request if you want.
These tools use the Python in the base conda environment since most of them need botocore for the S3 access.
Some tools help automate setting up the mc aliases for the filers (instead of a bunch of copy-pasting needed currently) and listing / copying files given filer paths (either given a regular filer path or a goofys mount path, but converts those into MinIO client paths and uses mc, doesn't depend on the goofys mounts at all). Makes using S3 directly / the MinIO client for things like uploading large files much easier than before.
Tests / Quality Checks
Are there breaking changes?
Ask yourself the next question;
If no, then carry on. If yes, there is a breaking change and we want to maintain the previous image do the following
Automated Testing/build and deployment
auto-deploytag to the PR before pushing in order to build and push the image to ACR so you can test it in cluster as a custom image?JupyterLab extensions
jupyter labextension listfrom inside the notebook)?VS Code tests
Code review
auto-deploytag to your PR before your most recent push to this repo? This causes CI to build the image and push to our ACR, letting reviewers access the built image without having to create it themselvesk8scc01covidacrdev.azurecr.io/jupyterlab-cpu:746d058e2f37e004da5ca483d121bfb9e0545f2b)?