Testing Tectonic on the arXiv paper dataset

[Mrmaxmeier]

I'm experimenting with a few low-level changes in the engine. Tectonic's test suite is a nice early warning but passing the tests says little about being able to build real LaTeX documents in my experience.
The arXiv archive provides TeX sources for most of the published papers.
I've hacked up a proof of concept of a crater-like regression testing tool using parts of the arXiv dataset.

Here's a quick rundown:
It'll run the engine on each source file and store logs and output files.
This takes about 30 minutes for 2500 papers.
It currently relies on libfaketime to get as close to reproducible builds as possible. Tectonic's output is now reproducible when setting SOURCE_DATE_EPOCH. This means we're able to detect regressions by checking the output hashes. Visual comparisons of output PDFs might still be interesting though in the future.
There are some issues with deterministic outputs as some samples trigger undefined behavior.
I've borrowed some of crater's CSS and whipped up a simple front-end for the test results.

I'll open separate issues for the false-positives caused by undefined behavior and will keep this one open as a kind-of tracking issue on reproducibility and regression testing.

Ideas / TODOs:

• valgrind / asan / msan
• visual diff for PDFs
• remove libfaketime hacks
• track performance
• run all samples in-process (might uncover some state corruption)
• automated runs for PRs (CI Bot Mrmaxmeier/tectonic-on-arXiv#5)

[pkgw]

I've wanted to do this for so long, this is amazing!!!

[pkgw]

Let me know if you're ever in the Boston area and I'll buy you the beverage(s) of your choice.

[XVilka]

Would be interesting to test also the Oxidized fork/branch:

• Oxidize everything #459
• https://github.com/crlf0710/tectonic/tree/oxidize

[Mrmaxmeier]

Would be interesting to test also the Oxidized fork/branch:

• https://github.com/crlf0710/tectonic/tree/oxidize

Here's the current results with crlf0710#122 applied.
https://tt.ente.ninja/#/compare/master-v0.1.11-326-g926360a/oxidize-v0.1.11-670-g8519cfc4-dirty

[pkgw]

@Mrmaxmeier Thank you, that's super helpful!

[burrbull]

404 on right column.

[cormacrelf]

Where are the sources for the test runner & frontend?

[burrbull]

On arxiv.org under Other formats link.

[cormacrelf]

@burrbull not the TeX sources, the code that runs the tests and displays them on https://tt.ente.ninja.

[Mrmaxmeier]

@cormacrelf
It's a collection of hacky python scripts and a web page that consumes .json reports:
https://github.com/Mrmaxmeier/tectonic-on-arXiv

It might make sense to rewrite this to consume tectonic as a library. It currently just spawn a bunch of tectonic processes.

[burrbull]

@Mrmaxmeier can you rerun tests with crlf0710#123 ?

[pkgw]

Now that these tests are running automatically in the main repo, I am going to say that this issue is closed. Thank you for you work to set up this service @Mrmaxmeier, it is hugely valuable!