Reports directory or XML files are not created

[sipani909]

I have added the following rake code:

require 'digest'
require 'ci/reporter/rake/minitest'
ENV["CI_REPORTS"] = "reports/"
.
.
.
.
desc 'Run Subset of Tests'
task :subset => 'ci:setup:minitest' do
.
.
.

But no reports folder or xml files are being generated.

ruby 2.6.3p62 (2019-04-16 revision 67580)
Minitest-5.14.4
ci_reporter_minitest-1.0.0

[sethpeters940]

I'm having the same problem as @sipani909 on ruby 3.0.0 and ci_reporter_minitest-1.0.0.

[hakanai]

Same here on ruby 3.2.0 but I'm not sure whether it's the Ruby upgrade or the Rails upgrade which did it, because we went through major versions of both in the process and didn't notice reports were missing until the very end.

I've done some investigation. Minitest does its best to prevent being investigated by setting reporter to nil as soon as it gets a reference for itself, so I had to jump through some hoops, writing my own plugin to steal the reporter at startup, and then looking at what reporters had been added to it by the time the test ran.

I see three reporters on it:

• a Minitest::SuppressedSummaryReporter - this also throws an exception from to_s so it must be Quality Software
  • Minitest::SuppressedSummaryReporter
  • Minitest::SummaryReporter
• a Rails::TestUnitReporter
  • Rails::TestUnitReporter
• a CI::Reporter::Runner

So the reporter seems to be present, but since this SuppressedSummaryReporter appears to be so broken that it can't print itself out as a string, perhaps it's also breaking the report chain somehow - is it possible getting one error in a minitest reporter will cause it to skip all subsequent reporters?

Minitest::CompositeReporter's implementation would appear to back this theory up:

https://github.com/minitest/minitest/blob/master/lib/minitest.rb#L881-L885

[hakanai]

It looks like there are at least two issues at play - the issue in minitest where some other plugin I had was throwing an error that stopped this plugin being run, but after removing the other plugin, ci_reporter also throws an error. Ironically for the exact same reason.

The issue with ci_reporter is already filed: ci-reporter/ci_reporter#167

The fix is also already committed, but there hasn't been a release in forever.