[Testing] Poor parallelism of test-roxygen-examples-complete.R

[MichaelChirico]

I think this file would best be split up into many files to support parallelism. As it stands, this test (test_that("analogous to test-roxygen-examples-complete")) takes 5x longer than any other test_that() units in the suite.

That means the shard stuck with this test is going to take much longer than all the others.

Ideally, each of these:

expect_warning(test_collection(
  "roxygen-examples-complete", "^1[^1234567890]",
  transformer = style_text
), NA)

would be in a separate file. I'm not sure the easiest way to accomplish this from an ongoing maintenance PoV.

NB: using different test_that() for each won't help since testthat parallelizes by file:

https://testthat.r-lib.org/articles/parallel.html

[lorenzwalthert]

Ok, was not aware of this. You can file a PR if you want to implement that.

[MichaelChirico]

what do you think is the best approach? just dump a dozen or so new files like test-roxygen-examples-complete-%02d.R?

[lorenzwalthert]

Can we make a folder and put the files in it or does testthat not support nested tests?

[MichaelChirico]

I was wondering the same thing & don't know offhand. let me look into it.

[MichaelChirico]

seems not supported; I commented our use case there:

r-lib/testthat#916

https://stackoverflow.com/q/48991329/3576984

ok to just proceed with splitting into many files in the meantime?

[lorenzwalthert]

Yes, I don't mind more files in the test directory, it already has quite a few. Maybe a reasonably small number, not sure every test hast to be in its own file for the best performance since a new file means also a new process has to be started with some overhead, no? And I don't think it makes sense to expect more than 16 cores anyways.

[MichaelChirico]

IIUC the full set of n files is batched into n/threads sets, so not sure about the overhead part.

my thinking for fully splitting was it's less arbitrary & so easier to maintain, but we could also batch in groups of say 4.

anyway we can continue from the PR.

[lorenzwalthert]

Thanks for the PRs. I think we first want to get #1103 since all test fail until we merge that.