Replies: 1 comment
-
Done as of 27c9b0f |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Help
Description
In highly parallel workloads, I almost always recommend
storage = "worker"
andretrieval = "worker"
so that the parallel workers manage the data instead of putting the burden on the main process. The default values for both are"main"
only because I envisioned cloud-based workloads where data may need to travel over the network. In practice, cloud pipelines almost always have some kind of object storage (or database storage) such as an S3 bucket. So I think I should set"worker"
as the default for both. But first, I would just like to check with those of you who follow the discussions to see if this would have any unintended consequences.Beta Was this translation helpful? Give feedback.
All reactions