Previous metadata for targets generated with format="file_fast" in v. 1.7.1 results in errors when upgrading to 1.8.0

[koefoeden]

Prework

• I understand and agree to help guide.
• I understand and agree to contributing guide.
• New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Hi!

I just updated targets from 1.7.1 to 1.8.0, and had found that all my previous targets with format="file_fast" failed to load/run without a very clear error message. The specific error message given is: "applicable method for 'store_class_format' applied to an object of class "c('file_fast', 'character')""

I managed to solve it by calling tar_invalidate on the targets. Looks like having previous metadata for targets with file_fast format does not play nice when updating to 1.8.0 (even continuing to use "file_fast" in 1.8.0 also errored - seems related to tar_load'ing the object when it has metadata, even if if the object itself is removed in the store).

This is the context in which it happened for me:

tar_target(name = YAML_path_from_GWAS_cat, 
 command = download_sumstats_or_YAML_file_from_GWAS_cat_path(GWAS_cat_FTP_link = GWAS_cat_FTP_link, type = "yaml"), 
 pattern = map(GWAS_cat_FTP_link), 
 format = "file"),
tar_target(name = sumstats_path_from_GWAS_cat, 
 command = download_sumstats_or_YAML_file_from_GWAS_cat_path(GWAS_cat_FTP_link = GWAS_cat_FTP_link, type = "sumstats"), 
 pattern = map(GWAS_cat_FTP_link), format = "file"),
tar_target(name = combined_finemapped_SNPs_from_GWAS_cat_paths, 
 command = get_finemapped_SNPs_from_GWAS_path(yaml_fname = YAML_path_from_GWAS_cat, 
 sumstats_fname = sumstats_path_from_GWAS_cat, trait=trait), 
 map(YAML_path_from_GWAS_cat, sumstats_path_from_GWAS_cat, trait), 
 resources = high_req, format = "file", iteration = "vector")

Thought you might be interested in this, as it was quite disruptive for me!

Traceback below:

base::tryCatch(base::withCallingHandlers({ NULL base::saveRDS(base::do.c...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), na...
    doTryCatch(return(expr), name, parentenv, handler)
    tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    base::withCallingHandlers({ NULL base::saveRDS(base::do.call(base::do.ca...
    base::saveRDS(base::do.call(base::do.call, base::c(base::readRDS("/scrat...
    base::do.call(base::do.call, base::c(base::readRDS("/scratch/RtmpEKqCvn/...
    (function (what, args, quote = FALSE, envir = parent.frame()) { if (!is....
    (function (targets_function, targets_arguments, options, envir = NULL, s...
    tryCatch(out <- withCallingHandlers(targets::tar_callr_inner_try(targets...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    withCallingHandlers(targets::tar_callr_inner_try(targets_function = targ...
    targets::tar_callr_inner_try(targets_function = targets_function, target...
    do.call(targets_function, targets_arguments)
    (function (pipeline, path_store, names_quosure, shortcut, reporter, seco...
    crew_init(pipeline = pipeline, meta = meta_init(path_store = path_store)...
    self$run_crew()
    self$iterate()
    self$process_target(queue$dequeue())
    self$run_target(name)
    if_any(target_should_run_worker(target), self$run_worker(target), self$r...
    self$run_main(target)
    target_conclude(target, self$pipeline, self$scheduler, self$meta)
    target_conclude.tar_builder(target, self$pipeline, self$scheduler, self$...
    builder_error(target, pipeline, scheduler, meta)
    builder_record_error_meta(target, pipeline, meta)
    meta$handle_error(record)
    self$get_record(record$name)
    record_from_row(row = self$database$get_row(name), path_store = self$store)
    store_path_from_record(store = store_mock(format = record$format, reposi...
    store_mock(format = record$format, repository = record$repository)
    store_enclass(mock, format = format, repository = repository)
    store_class_format(store_dispatch_format(format))

[koefoeden]

It can potentially also be related to the fact that two different instances (one full, and one data-subsampled test instance) of the pipeline, i.e. with different stores for objects & metadata, have written to these same files before.

If you by the way have any tips/suggestions on how to best go about this case (having the ability to interchangably run all targets with smaller test data sets and the full dataset, without one instance invalidating the other) that would also be greatly appreciated. I tried to implement it through branching, but it quickly became unmanagable for my large pipeline.

[wlandau]

Should be fixed in the 728d2dc. You can either install the development version

remotes::install_github("ropensci/targets")

or, to continue using the current release version (1.8.0), you could replace every occurrence of "|file_fast|" with "|file|" in the _targets/meta/meta file.