Pipeline hangs upon erroring when workspace_on_error='trim' (i.e doesn't quit if no more reasonable targets are able to run)

[koefoeden]

Prework

• I understand and agree to help guide.
• I understand and agree to contributing guide.
• New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Hi!

I was super excited to use the workspace_on_error="trim" parameter, as I think it makes a lot of sense. However, I have experienced that the pipeline simply hangs after encountering an error when this is enabled. The fact that it does not appropriately quit (even if there are no more targets to reasonably run), makes it quite hard to debug, as there doesn't seem to be any error messages logged in the usual places.

It seems to only be an issue when using the crew worker-framework - and both with and without running via the Slurm-plugin.

Here is my setup, using a controller-group:

low_req_ctrlr <- crew::crew_controller_local(name = "low_req_ctrlr_name", workers = 100, seconds_timeout = 600)
 mid_req_ctrlr <- crew::crew_controller_local(name = "mid_req_ctrlr_name", workers = config_n_workers, seconds_timeout =600)
 high_req_ctrlr <- crew::crew_controller_local(name = "high_req_ctrlr_name", workers = 2, seconds_timeout = 600)

targets::tar_option_set(
    storage = "worker",retrieval = "worker", deployment="worker",
    controller = crew::crew_controller_group(low_req_ctrlr, mid_req_ctrlr, high_req_ctrlr),
    resources = targets::tar_resources(crew = targets::tar_resources_crew(controller = "mid_req_ctrlr_name")))

[koefoeden]

This is the output of the pipeline, where it just hangs on recorded workspace (if this feature is enabled - but also hangs without recording workspace) - until it is manually stopped. It looks like it keeps a single-process running indefinitely, utilizing 100 % CPU of one CPU core.

▶ dispatched target so_w_chromvar
▶ recorded workspace so_w_chromvar
^C
Execution halted

[koefoeden]

Two additional notes:

1. When "trim" is enabled (without using crew, so that it appropriately quits), it seems that the normal debug/error message is also skipped.

2. The new error message, when not using trim, also seems to use the wrong target name. The actual "Last error message" seems to be correct though. Curiously, as far as I recall, it is the same target each time, independent of where the crash happens in the pipeline. See an example below:

▶ dispatched target so_w_chromvar
▶ recorded workspace so_w_chromvar
✖ errored target so_w_chromvar
✖ errored pipeline [2.793 minutes]

Error:
! targets::tar_make() error

── Debug target sum_exp_w_pseudobulk_peaks_final ─────────────────────────
tar_meta(sum_exp_w_pseudobulk_peaks_final)$error
tar_workspace(sum_exp_w_pseudobulk_peaks_final)

── General debugging ───────────────────────────
• tar_meta(fields = any_of("error"), complete_only = TRUE)
• tar_workspace()
• tar_workspaces()

── How to ─────────────────────────
• Debug: https://books.ropensci.org/targets/debugging.html
• Help: https://books.ropensci.org/targets/help.html

── Last error message ─────────────────────
the rownames and colnames of the supplied assay(s) must be NULL or
identical to those of the RangedSummarizedExperiment object (or
derivative) to construct

── Last error traceback ────────────────
add_chromvar_assay(so_w_clusters)
Signac::RunChromVAR(object = so, genome = BSgenome.Hsapiens.UCSC.hg38::B...
RunChromVAR.Seurat(object = so, genome = BSgenome.Hsapiens.UCSC.hg38::BS...
RunChromVAR(object = object[[assay]], genome = genome, motif.matrix = mo...
RunChromVAR.ChromatinAssay(object = object[[assay]], genome = genome, ...
SummarizedExperiment::SummarizedExperiment(assays = list(counts = peak.m...
stop(wmsg("the ", what, " of the supplied assay(s) must be NULL ", ...
.handleSimpleError(function (condition) { state$error <- build_mess...
h(simpleError(msg, call))
Execution halted

[wlandau]

Would you please share a runnable reproducible example?

[wlandau]

1. When "trim" is enabled (without using crew, so that it appropriately quits), it seems that the normal debug/error message is also skipped.

That's by design because the pipeline allows more than one target to error out under non-default error settings.

[wlandau]

I think I reproduced it with:

library(targets)
tar_option_set(
  error = "trim",
  controller = crew::crew_controller_local(workers = 2)
)
list(
  tar_target(index, seq_len(2L)),
  tar_target(a, {
    if (index == 1L) {
      index
    } else {
      Sys.sleep(5)
      stop(index)
    }
  }, pattern = map(index)),
  tar_target(b, a, pattern = map(a)),
  tar_target(c, b, pattern = map(b)),
  tar_target(d, a, pattern = map(a)),
  tar_target(e, d, pattern = map(d))
)

[wlandau]

I think I fixed the original deadlock issue with error = "trim".

2. The new error message, when not using trim, also seems to use the wrong target name. The actual "Last error message" seems to be correct though. Curiously, as far as I recall, it is the same target each time, independent of where the crash happens in the pipeline. See an example below:

I haven't been able to reproduce this. If you can create a small runnable example that reproduces it, please share in a new issue.

Using:

library(targets)
list(
  tar_target(index, seq_len(2L)),
  tar_target(a, {
    if (index > 1L) {
      stop(index)
    }
  }, pattern = map(index)),
  tar_target(b, a, pattern = map(a)),
  tar_target(c, b, pattern = map(b)),

  tar_target(d, a, pattern = map(a)),
  tar_target(e, d, pattern = map(d))
)

I see a consistent target name in the messages:

library(targets)
tar_make()
✔ skipped target index
▶ dispatched branch a_ef1ae9b9954fc770
● completed branch a_ef1ae9b9954fc770 [0 seconds, 43 bytes]
▶ dispatched branch a_6d03443cebc2cf01
▶ recorded workspace a_6d03443cebc2cf01
✖ errored branch a_6d03443cebc2cf01
✖ errored pipeline [0.104 seconds]
Error:
! targets::tar_make() error

── Debug target a_6d03443cebc2cf01 ───────────────────────────────────────────────────
    tar_meta(a_6d03443cebc2cf01)$error
    tar_workspace(a_6d03443cebc2cf01)

── General debugging ─────────────────────────────────────────────────────────────────
    • tar_errored()
    • tar_meta(fields = any_of("error"), complete_only = TRUE)
    • tar_workspace()
    • tar_workspaces()

── How to ────────────────────────────────────────────────────────────────────────────
    • Debug: https://books.ropensci.org/targets/debugging.html
    • Help: https://books.ropensci.org/targets/help.html

── Last error message ────────────────────────────────────────────────────────────────
    2

── Last error traceback ──────────────────────────────────────────────────────────────
    stop(index)
    .handleSimpleError(function (condition)  {     state$error <- build_mess...
    h(simpleError(msg, call))

[koefoeden]

Hi Will

Thanks for looking into this! Unfortunately I still seem to experience the deadlocks when using the "trim" option with local crew-controllers and the newest development version. My pipeline is also quite complex using multiple controllers (i.e. controller groups), so it takes a lot of time to designing/figuring out a reprex. Let me know if you have any tips/intuition on what could be the issue, that could potentially guide me in designing the reprex. Otherwise, I'll get back to you when I've had the chance to dig deeper into it.