Defaults for the storage, retrieval, and memory settings

[wlandau]

Discussed in #1372

^{Originally posted by wlandau November 10, 2024}

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

In highly parallel workloads, I almost always recommend storage = "worker" and retrieval = "worker" so that the parallel workers manage the data instead of putting the burden on the main process. The default values for both are "main" only because I envisioned cloud-based workloads where data may need to travel over the network. In practice, cloud pipelines almost always have some kind of object storage (or database storage) such as an S3 bucket. So I think I should set "worker" as the default for both. But first, I would just like to check with those of you who follow the discussions to see if this would have any unintended consequences.

[wlandau]

Thinking about this for the next release.

[wlandau]

Adding transient memory to this thread too. storage = "worker", retrieval = "worker", and memory = "tranisient" are settings I have recommended in https://books.ropensci.org/targets/performance.html for years. Through user feedback and direct experience, these seem to be the right defaults when performance matters. I will change them for the next release.

[wlandau]

I forgot about memory = "auto", the current default. This should usually be efficient when retrieval is "main" and many dynamic branches are loading a non-dynamic target, but we'll mostly want retrieval = "worker" anyway.

[wlandau]

On reflection, memory = "persistent" and retrieval = "worker" could be problematic for dynamic branching in common cases. Consider:

list(
  tar_target(data, get_big_data(), memory = "persistent"),
  tar_target(models, model_data(data), pattern = map(data), retrieval = "worker")
)

Each branch of models would load the entire data target into memory on the worker. For heavily branched workflows, this would read the same file from disk an excessive number of times.

The default memory setting should be "persistent" if the target is a stem (lacks a pattern argument) and another target dynamically branches over it.

The default retrieval setting should be "main" if the target branches over a stem.