Bottlenecks in tar_make() if targets are up to date

[wlandau]

Similar to #1458. In the following sequential pipeline:

library(targets)
list(
  tar_target(datasets, seq_len(1e5), memory = "persistent"),
  tar_target(models, datasets, pattern = map(datasets), retrieval = "main")
)

proffer::pprof(tar_make(callr_function = NULL)) takes 2.4 minutes and shows:

[wlandau]

On reflection, there's not much of a reason to have junction_extract_index(). We just need to store the index inside the branch reference objects. It's just an unnamed integer scalar, so indexes shouldn't take up too much more memory in aggregate.

[wlandau]

After removing junction_extract_index(), it only takes 1.9 minutes, and the profiling data looks better:

[wlandau]

In meta$hash_deps(), the bottleneck is unsurprisingly the loop that digs for individual branch hashes:

targets/R/class_meta.R

Lines 91 to 95 in 175a621

	while (index <= n) {
	name <- .subset(deps, index)
	hashes[[name]] <- .subset2(.subset2(lookup, name), "data")
	index <- index + 1L
	}

For dynamic branches, we could make that more efficient by storing the branch hashes in a fixed-length character vector in the junction object.

[wlandau]

Turns out the slowness comes from incrementally building a 10000-element hashes list element by element. caa7cf5 converts this to lapply() and is much faster (down to 44 seconds):