Use or remove dh_auto_test

[trainman419]

Currently, dh_auto_test runs the tests for every package, and then ignores the results.
See rules.em line 45

For about half of all packages, this does nothing because they have no tests. On most of the remainder, it wastes a significant amount of time on my build farm. In a few rare cases, the tests hang and eventually cause the package build to fail.

If running tests isn't valuable, they shouldn't be run. If running the tests is valuable, we shouldn't ignore the results.

[wjwwood]

Yeah, I noticed this before. I don't remember the rational behind running the tests always. I think at least for building the tests someackages dont properly support the tests not being built. Though that might be untealted to this. @dirk-thomas and @tfoote would know better than me.

[tfoote]

The goal was to make the tests matter and gate releaeses. However since we've determined that the test dependencies are not the same as the build dependencies we can't guarantee they will even build. So we should probably just disable them entirely.

[wjwwood]

Can you guys review: #293

[wjwwood]

Another option is to figure out if there is an option on the farm to prevent dh from executing the test stage all together. This would optimize existing bloom'ed packages as well.

[dirk-thomas]

@tfoote The dependencies (test vs. build) are not related to this at all. The jobs can already do the following:

• install build dependencies
• build
• install test dependencies
• build and run tests

Even for package which do not support CATKIN_ENABLE_TESTING correctly there are CMake options in catkin to make them "behave".

Imo release jobs with unit test failures should be flagged by Jenkins as "unstable" at least.

Having failing unit tests is not desirable but sometimes just the case (e.g. when adding a new test for a bug before being able to fix it). But having the feedback that you actually release a unstable package would be a good thing imo.

[wjwwood]

Ok, so another option is to keep tests, but use them somehow.

That will require changes in the build farm I think, but I am ok with both. Let me know how bloom should change if we want to do that.

[trainman419]

On the topic of disabling tests, the documentation for dh_auto_test indicates that setting DEB_BUILD_OPTIONS=nocheck in the environment will disable tests, bu this doesn't appear to be working on my farm, or it isn't making it all the way to dh_auto_test inside of pbuilder.

[trainman419]

We could also switch the || true with something like || echo "TESTS_FAILED. Continuing anyway" and then setting up a build trigger to catch that.

Unfortunately, that doesn't solve my issue, where certain tests hang forever and must be killed.

[wjwwood]

@trainman419 that's why I suggested looking into an option which can be passed to dh by the build farm, then it can be a build farm configuration. If you disable them in bloom, then you have no easy way to enable them in the future.

[tfoote]

It looks like if you override dh_auto_test you need to check for the flag: https://bugs.launchpad.net/python-cinderclient/+bug/1098645

This is the recommend approach from debhelper ticket: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=568897

Somebody wanted to have a warning if that didn't happen but it looks like it was too complicated to do: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=712394

[trainman419]

We're overriding the dh_auto_test target using dh_auto_test, which should be checking the DEB_BUILD_OPTIONS environment variable, so I don't think we have to worry about adding an extra check for it.

I suspect the real reason that setting DEB_BUILD_OPTIONS on my system isn't working is because pbuilder isn't passing the external environment into the chroot.

[tfoote]

@trainman419 Independent of whether it's getting passed in, it doesn't automatically check for the flag. That needs to be added to our override.

[wjwwood]

What is the conclusion of this discussion?

[tfoote]

We want to make this equivilant change: https://bugs.launchpad.net/python-cinderclient/+bug/1098645

Such that we check for the DEB_BUILD_OPTIONS in the override_dh_auto_test