bottlenecks in tar_outdated() for large projects

[wlandau]

I wonder how feasible it is to remove these findings. Might be unfeasible, but still worth looking at again:

[wlandau]

Progress: 0ab09a5 reduced target_skip() down to 5.13% of the runtime of tar_outdated().

[wlandau]

Recent commits (up to e026925) have reduced the computation time of tar_outdated() by about 33%. I still think I can eliminate calls to meta$get_record() by relying on direct metadata rows instead of full record objects, especially in target_sync_file_meta().

[wlandau]

Flame graph from proffer::pprof(targets::tar_outdated(callr_function = NULL, reporter = "silent")) as of bc09d7a:

I think builder_should_run() and pipeline_get_target() are worth revisiting next.

[wlandau]

As of acbb8b5, tar_outdated() runs about 4.6 times faster than the current CRAN version. Results from a real-world pipeline with over 66000 targets:

version	elapsed seconds (1.9.1)	elapsed seconds (1.9.1.9005)	speedup
Mac OS	115.443	24.986	4.620
RHEL9	252.349	53.939	4.678

Flame graph on Mac OS:

Flame graph on RHEL9:

Improvements could be made still: for example, target_update_depend() is entirely overhead and could be reengineered. But I think this progress on optimization is good enough for now.