Input files packaging

[andrii-i]

When working with notebooks, users frequently import and use additional files such as datasets, images, and scripts inside their notebook's cells. Providing support for packaging such files would ensure that notebooks can have all essential resources available when executed as a job. This would make Jupyter Scheduler more flexible and able to accommodate more types or workflows providing better value to people who use it.

This PR adds an option to package input folder (folder where input notebook is located) and all nested files and sub-folders within it during the job or job definition creation.

In terms of features, this is a subset of PR #500. This PR does not automatically download output files to output folder when job runs and therefore has no need need to schedule downloads from multiple processes and components that would manage it (DownloadRunner and DownloadManager from #500). Besides making this PR more focused in terms of functionality, this makes changes introduced by this PR non-breaking.

When package input folder option is active:

• All files and sub-folders within input folder are copied to the staging area together with the snapshot of an input notebook
• Original organization of files and folders (file tree) is maintained, input notebook is able to access them via relative file paths when it is ran as a job
• After downloading job files, users can access the output folder by clicking "Files" shortcut under "Output files" in the list view (it will open the folder in JupyterLab filebrowser).
• For jobs, packaged input files and side effects are tracked in Job.packaged_files, if any of them is deleted from output folder, user gets an option to re-downloaded them via UI (matches existing behavior for snapshot of the input notebook and output files)
• For job definitions, packaged input files and side effects are tracked in JobDefinition.packaged_files
• Modifies notebook / job execution logic so that kernel executing the notebook always operates in the same directory as the notebook being executed (cwd parameter of the ExecutePreprocessor). Additionally pass intended path of execution context to preprocessor via metadata {"metadata": {"path": notebook_dir}} argument of the preprocessor call.
• Side effects of running an input notebook are added to Job.packaged_files and copied to the output folder together with other files

Fixes #407

Before:

After:

Re-download option when any of the files is deleted from the output folder:

[andrii-i]

Kicking CI

[andrii-i]

Kicking CI

[andrii-i]

Kicking CI

[andrii-i]

Could you please update the screenshots and text in the docs to reflect this new option? I can help with the copy editing.

@JasonWeill updated "Submit the Create Job form" section of the Users page with a screenshot and text mentioning "Run job with input folder" option (code, readthedocs preview).

I tried different options like adding an example "Use this to, for example, access data files or images from notebook's cells." But ultimately what I have in the PR now works well and matches level of detail given in the paragraphs around it, would be interested to learn your opinion on this.

[andrii-i]

Kicking CI

[JasonWeill]

Change looks good! Thanks for the great work on this.