Integration tests create stub modules in app/code

[magento-engcom-team]

When even a single integration test is executed, the following files and directories are created:

app/code/Magento
├── TestModuleDirectoryZipCodes
│   ├── etc
│   │   ├── module.xml
│   │   └── zip_codes.xml
│   └── registration.php
├── TestModuleFakePaymentMethod
│   ├── Gateway
│   │   └── Command
│   │       └── DoNothingCommand.php
│   ├── etc
│   │   ├── config.xml
│   │   ├── di.xml
│   │   └── module.xml
│   └── registration.php
└── TestModuleSample
    ├── composer.json
    ├── etc
    │   └── module.xml
    └── registration.php

(Note: the module Magento_TestModuleFakePaymentMethod is not generated in Magento 2.2.0).
This is a problem because running integration tests should not affect the installed Magento instance.

Preconditions

Reproduced on fresh installations of Magento 2.2.0 and 2.2.2. I haven't tried on Magento 2.2.1.

Steps to reproduce

1. Install Magento, e.g. composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition . and so on.
2. Run all integration tests or create a custom module with nothing but a single integration test.
3. Run the test, e.g. /var/www/example/vendor/phpunit/phpunit/phpunit --configuration /var/www/example/dev/tests/integration/phpunit.xml /var/www/example/app/code/Example/Foo/Test/Integration

Expected result

1. Only the specified tests should run in isolation.
2. Potential production code should not be affected.

Actual result

The modules are created in app/code/Magento potentially affect production code and are created independently if they are used in any test.

It should be possible to create the modules under dev/tests/integration/tmp in the sandbox directory and register them in the test process with the component registrar instead, leaving the app/code directory unmodified.

Original Report: magento/magento2#12696 by @Vinai

[Vinai]

Well... I would classify this as a bug, not a feature request, but this is not my choice to make.
Cheers :)

[miguelbalparda]

@magento-engcom-team this is a bug report, not a feature request. I don't have privileges over this repo, maybe you can close this one?

[astyczen]

@Vinai , @miguelbalparda has this been delt with in some other issue (as a bug)? I can still see this happen in Magento 2.3.0 for Amqp module tests and others.

[Vinai]

The number of modules that get created during an integration test run has increased (a lot) instead of decreased. @schmengler and @avstudnitz have worked on a workaround during a contribution day where the modules are deleted again after execution, but I think that is not a proper solution.

It should be possible to create the modules somewhere in the integration test sandbox directory, add them to the component registrar, reconfigure the test ObjectManager so their configurations in di.xml are applied, execute the tests and then remove the test modules again.
Reconfiguring the ObjectManager after module creation and brefore test execution and during teardown is probably the most challenging task, but certainly not impossible.

[avstudnitz]

@astyczen You can find the workaround Pull Request here: magento/magento2#18459

[astyczen]

@Vinai , @avstudnitz thank u for information, I will check out the solution from the PR.