External databases for metadata management

[wlandau]

targets uses simple text files for metadata (pipe-separated values). These files can get large in pipelines with many targets (#1390) and appending to them creates overhead in tar_make(). Maybe targets can instead use a DuckDB database for metadata.

[r2evans]

Side note for this: if you have more than the main process writing to the database, you will need to deal with the access-contention. DuckDB supports either multiple readonly readers, or only one writeable connection, you cannot have one connection write while any other connection is reading. See Concurrency.

[wlandau]

Thanks for pointing that out. This may be a deal-breaker for metadata. Redis seems like another good option if the profiling data shows a speedup. Though I don't think I'd make Redis the default because it can be a burden to install.

[r2evans]

I have considered duckdb for many larger-scale projects where having a local quasi-permanent store would be good, but the concurrency issue is a deal-breaker, and not one they seem eager to remedy, unfortunately. Redis would be easy enough for many, but I agree that even as easy as it is, running a redis instance "just for this" might be more than some people prefer.

If you're considering Redis but don't want the server overhead (and perhaps would like on-disk persistence), rrlite can be a good option. Not just because it doesn't require a server, but also because the difference between rrlite and redux is just the initial connection ... otherwise it's intended to be API-compatible with redux, except file-based (or :memory:-based).

I haven't benchmarked it nor verified its concurrency (other than finding seppo0010/rlite#13 (comment)). Admittedly, it hasn't seen commits in many years, I don't know if that means it's awesome-stable or not. richfitx is the author/maintainer (also maintains redux).

[wlandau]

I have been profiling example pipelines, and the bottleneck seems to be reopening the base::file() connection for each line appended to _targets/meta/meta. It would be much faster to keep a persistent open connection, but it may interfere with other connections that read the file. I will test.

[wlandau]

Maintaining a persistent connection seems to reduce execution time from around 60 seconds down to around 23 seconds on an M2 Mac in the following 10000-target pipeline:

library(targets)
tar_option_set(
  controller = crew::crew_controller_local(workers = 25L)
)
list(
  tar_target(datasets, seq_len(1e4), memory = "persistent"),
  tar_target(models, datasets, pattern = map(datasets), retrieval = "main")
)

[wlandau]

There's still a ~30% bottleneck in cat() when appending though.

[wlandau]

Negligible improvement with file(blocking = FALSE).

[wlandau]

There's still a ~30% bottleneck in cat() when appending though.

This is only when targets complete instantaneously.

Moving to a database may be necessary, but on reflection, it is a bit extreme. Converting to a discussion.