Use available streams support functions in pipeline execution #1777
Description
Stemming from #1690, Depends on #1703
The most direct way to fix this would probably be to update the storage adapter API so getFile returns a ReadableStream rather than a Buffer. Similarly, putFile should probably accept a ReadableStream for the file input (or perhaps adding support for a ReadableStream but not requiring it).
Taking a quick look at the codebase, the following files would need to be updated:
- common/plugin/LocalExecutor.js
- common/storage/index.js (Add stream support in storage clients (Closes #1702) #1703)
- common/plugin/GeneratedFiles.js
- plugins/UploadArtifact/UploadArtifact.js (Won't be necessary after Bypass blob storage in UploadArtifact(Closes #1704) #1717)
- common/storage/backends/StorageClient.js (Add stream support in storage clients (Closes #1702) #1703)
- plugins/GenerateJob/templates/run-debug.js
- plugins/GenerateJob/templates/start.js
- routers/InteractiveCompute/job-files/start-session.js
- gme client (Add stream support in storage clients (Closes #1702) #1703)
- sciserver-files client (Add stream support in storage clients (Closes #1702) #1703)
- s3 client (Add stream support in storage clients (Closes #1702) #1703)
Originally posted by @brollb in #1690 (comment)