[SUGGESTION] Basic Continuous Integration

[jarzec]

The statement
This is a kind of meta-suggestion. The goal is not to provide a language feature request for the experimental new syntax of C++ itself but to streamline the development of all of them.

I propose to add basic scripts that would automatize running cppfront through all the existing tests and would not complicate addition of new test cases. As this should only serve utility purposes not the main goal of the experiment, it should be kept to the minimum. To allow automated execution of existing and future tests following their original idea it is enough to add just basic scripts (one for Windows and one for Linux). The scripts would allow to run the tests for all the compilers targeted by the project. An example of such script for Linux can be seen in this commit. This is a variation on the existing regression-tests/run-tests.bat that already exists in the project. The proposition takes it a step further and performs comparisons against the expected outputs.

Such scripts would allow for quick and easy verification of any proposed change in the project's source code. With the addition of checks against available reference code/output a trivial Github workflow can be defined to automatically execute tests for any PR. Such automated workflow can be added to the project without any extra configuration effort, just by adding a single file. This should greatly facilitate evaluation of change/fix propositions.
An example can be seen in the commit mentioned above.
The following PRs shows the effect of including the proposition in the project:

• Here is an example of a PR that introduces breaking changes both in the code generated by cppfront and in its output. The goal is to show how test failures can be reported on the PR website as well as how failure details can be presented.
• Here is an example of a PR that introduces non-breaking changes.

To add new test with the proposed approach it is enough to add a .cpp2 source file in the regression-test directory and the respective .cpp and .output files in regression-test/test-results. Updating exisiting tests and expected results is equally trivial.

Alternatives
Test automation is clearly not essential for the goals of the experiment. As such this suggestion can be simply disregarded. However, few programmers (if any) would question the added value of CI tools (even the most basic ones) in terms of developability of any software and cppfront happens to be software.
There are many approaches that could be taken to automated testing that might involve build tools like CMake, Make, Ninja, you name it. My proposition is to keep it to the bare minimum. This seems to be the general spirit of the project: focus on the main goal of improving C++. Potential contributor should not need to install and learn any extra software. Using shell languages built into the OS seems to be the most convenient and the least cumbersome way to go for reasons mentioned above. It is also the one introducing the least of personal taste.

[hsutter]

Closed with #151.