Split the community build into two jobs on the CI to speed it up?

[smarter]

The community build is now taking about an hour to run, it would be nice if we could split it in two jobs to reduce the CI wait time. This is tricky since there's dependencies between various projects, perhaps the easiest split would be between lihaoyi's libraries and everything else, since lihaoyi's library tend to only depend on other lihaoyi's libraries, but I don't know if there's enough lihaoyi's libraries for this to be a good split. What do you think @anatoliykmetyuk ?

[nicolasstucki]

We might reduce CI time by a bit less if we manage to split it in a balanced way.

@anatoliykmetyuk what is the status on this?

[liufengyun]

Another thing that might improve throughput is to cancel running jobs for previous commits if we push a new commit to a PR.

Not sure if Github actions support this. There is something related:

• [1] https://github.community/t/is-it-possible-to-cancel-previous-jobs-automatically-when-the-pull-request-is-updated/16779/4
• [2] https://github.com/marketplace/actions/cancel-workflow-action

[nicolasstucki]

But only if the commit is not in the branch. Subsequent commits that are in the current history should all run if pushed individually.

[nicolasstucki]

Cancel Workflow Action seems to cancel all runs in progress. It will probably not work. Maybe it can work with some advanced setting.

[anatoliykmetyuk]

I tried splitting the CB in #9652. But it didn't help: each half of the CB still took 50 minutes to run:

https://github.com/lampepfl/dotty/runs/1036016282
https://github.com/lampepfl/dotty/runs/1036016288

[nicolasstucki]

Both those halfs ran scalatest.

[nicolasstucki]

Both seem to have run all the tests

[smarter]

One approach if there's a lot of dependencies and it's hard to split things up would be to ditch junit, make one sbt task per project, declare dependencies between these tasks, and rely on sbt to run things in parallel.