Add an API to do some customization and then directly run tests in process

[natebosch]

Background

Most of our runners assume some separation between the process that orchestrates running all the tests, and the tests themselves. This is different Isolates in the VM case, and external processes like browsers in other cases.

We also support running a _test.dart file directly:

test/pkgs/test_core/lib/test_core.dart

Lines 34 to 72 in 1c29a64

	/// The global declarer.
	///
	/// This is used if a test file is run directly, rather than through the runner.
	Declarer? _globalDeclarer;
	/// Gets the declarer for the current scope.
	///
	/// When using the runner, this returns the [Zone]-scoped declarer that's set by
	/// [IsolateListener] or [IframeListener]. If the test file is run directly,
	/// this returns [_globalDeclarer] (and sets it up on the first call).
	Declarer get _declarer {
	var declarer = Declarer.current;
	if (declarer != null) return declarer;
	if (_globalDeclarer != null) return _globalDeclarer!;
	// Since there's no Zone-scoped declarer, the test file is being run directly.
	// In order to run the tests, we set up our own Declarer via
	// [_globalDeclarer], and schedule a microtask to run the tests once they're
	// finished being defined.
	_globalDeclarer = Declarer();
	scheduleMicrotask(() async {
	var suite = RunnerSuite(const PluginEnvironment(), SuiteConfiguration.empty,
	_globalDeclarer!.build(), SuitePlatform(Runtime.vm, os: currentOSGuess),
	path: p.prettyUri(Uri.base));
	var engine = Engine();
	engine.suiteSink.add(suite);
	engine.suiteSink.close();
	ExpandedReporter.watch(engine, PrintSink(),
	color: true, printPath: false, printPlatform: false);
	var success = await runZoned(() => Invoker.guard(engine.run),
	zoneValues: {#test.declarer: _globalDeclarer});
	if (success == true) return null;
	print('');
	unawaited(Future.error('Dummy exception to set exit code.'));
	});
	return _globalDeclarer!;
	}

This support is hardcoded to the expanded reporter and does not take any arguments for things like test filters.

In all cases when we run tests through a runner (internally, flutter_test, build_test, and package:test runner) we "wrap" the users library with out own and forward to their main in some way where the details vary by platform.

Goal

We should make it easier to combine the concepts of running without an external runner doing the orchestration, along with allowing some customization beyond the defaults. The immediate need is to register and pick a custom reporter.

Plan

Add a function in package:test_core/backend.dart like Future<bool> runTests(FutureOr<void> Function() userMain, {ReporterDetails reporter}).

Then platforms that want to build a custom "runner" that is baked in to a single execution instead of having a separate orchestration would use it by generating a wrapping library like:

import 'package:test_core/backend.dart';
import 'package:my_custom_test/my_custom_test.dart' as custom;  // The reporter implementation.

import '$userTestFile' as test; // Generated portion of the file - imports the users _test.dart file

void main() {
  runTests(test.main, reporter: custom.reporter);
}

We'd start with ReporterDetails for now, and we can hopefully add other named arguments for other customizations that crop up. For instance we might support test filtering by tag with an additional named argument.

[natebosch]

We want to explore a bit on #1311 before making decisions about making more of this stuff public and spreading use.

[natebosch]

Another thing to consider here is whether and how we can support coverage.

@grouma - do you know if it is feasible to enable coverage collection from within the isolate?

[grouma]

Not sure what you mean by "within the isolate". However, package:coverage supports collecting coverage information for a given isolate ID. See this.

[natebosch]

Another approach to consider here, we could also have an API that accepts List<String> args and embeds the full arg parsing that the normal test runner allows. There would be the option of calling apis like registerReporter and to perform some manipulation of the arguments before forwarding them. The upside is that arg parsing doesn't need to be rewritten, the downside is that it adds a programatic dependency on being able to do specific arg manipulations which may be difficult to keep working across versions and it's not as clear. Also many of the arguments probably don't make any sense in this world - you likely won't really be using multiple "platforms", and there isn't going to be any real scanning for tests to run, it would also be the single library.

[natebosch]

Took a bit more look at the idea of handling this with List<String> args and I realized that it would force some platform specific imports on us, and I'm not sure we want this API to be limited by platform. Will keep running with allowing the Reporter as the sole configuration for now, but soon we'll need to figure out how to handle test filtering. See #1328 - a likely interaction mode here is going to be to run once to collect test IDs, and then run again once per test to invoke the individual tests. There is a lot of overhead there but it may be required.