dev:tests:run ignores functional tests

[kassner]

Hi,

Functional tests are not listed as a test type to run with bin/magento dev:tests:run:

magento2/app/code/Magento/Developer/Console/Command/DevTestsRunCommand.php

Lines 115 to 147 in 57f5ab8

	private function setupTestInfo()
	{
	$this->commands = [
	'unit' => ['../tests/unit', ''],
	'unit-performance' => ['../tests/performance/framework/tests/unit', ''],
	'unit-static' => ['../tests/static/framework/tests/unit', ''],
	'unit-integration' => ['../tests/integration/framework/tests/unit', ''],
	'integration' => ['../tests/integration', ''],
	'integration-integrity' => ['../tests/integration', ' testsuite/Magento/Test/Integrity'],
	'static-default' => ['../tests/static', ''],
	'static-legacy' => ['../tests/static', ' testsuite/Magento/Test/Legacy'],
	'static-integration-php' => ['../tests/static', ' testsuite/Magento/Test/Php/Exemplar'],
	'static-integration-js' => ['../tests/static', ' testsuite/Magento/Test/Js/Exemplar'],
	];
	$this->types = [
	'all' => array_keys($this->commands),
	'unit' => ['unit', 'unit-performance', 'unit-static', 'unit-integration'],
	'integration' => ['integration'],
	'integration-all' => ['integration', 'integration-integrity'],
	'static' => ['static-default'],
	'static-all' => ['static-default', 'static-legacy', 'static-integration-php', 'static-integration-js'],
	'integrity' => ['static-default', 'static-legacy', 'integration-integrity'],
	'legacy' => ['static-legacy'],
	'default' => [
	'unit',
	'unit-performance',
	'unit-static',
	'unit-integration',
	'integration',
	'static-default',
	],
	];
	}

Is there any reason for this or was just forgotten?

Thanks.

[mazhalai]

@kassner AFAIK It is intentional: Functional tests have added dependency on MTF framework and Selenium (https://github.com/magento/magento2/blob/develop/dev/tests/functional/composer.json)

[kassner]

@mazhalai

There is any documentation for extension developers to create their own functional tests?

[mazhalai]

Introduction to Magento Testing Framework

[kassner]

@mazhalai

Unfortunately that link does not make clear how to run functional tests, not even to test the Magento itself.

I was able to run some of Magento's tests, but almost all of them result in an exception.

What I've currently done so far inside dev/tests/functional:

• composer install;
• Filled up etc/config.xml with application and database information;
• Filled up phpunit.xml.dist with app_frontend_url and app_backend_url;
• Started the Selenium jar manually in another shell;
• Ran vendor/bin/phpunit

My result, except for one test that registered a customer and logged in, I got:

• Tests failed with an exception: Application is not installed yet;
• Firefox opens and closes several times but does not navigate to any page, keeps in about:blank (Firefox's default page);

I'll keep digging. I appreciate any help.

Thanks.

[lanken]

Currently public documentation creation is in progress.

Tests failed with an exception: Application is not installed yet;

Such error can be occured only if Magento application isn't installed. Is a Magento successfully opened by the url that you have entered in app_backend_url?

Also, please, pay attention that in order to run functional tests in Backend you need to disable secret key in Magento (http://devdocs.magento.com/guides/v2.0/mtf/mtf_installation/mtf_preinstall.html)

[kassner]

@lanken Thanks for the update.

I've fixed this particular exception in my environment, but now I'm issuing a new one:

1) Magento\Catalog\Test\TestCase\Category\CreateCategoryEntityTest::test with data set "CreateCategoryEntityTestVariation3" (CreateCategoryEntityTestVariation3)
ReflectionException: Class Magento\Mtf\Handler\HandlerFactoryDeprecated does not exist

As far as I looked into the history, \Magento\Mtf\Factory\Factory was changed to use this "Deprecated" handler back in January. I even tried to use tags before this (1.0.0-rc16 and 1.0.0-rc15), use the develop branch and use the \Magento\Mtf\Handler\HandlerFactory class, but all of those options lead me to some other errors like missing methods or autoloading issues.

I'm not sure why this file is missing (or why the call was renamed to "Deprecated"), I'm hoping I'm using the wrong repo (I'm using this one). Also, http://packages.magento.com/ does not have anything about mtf.

I'm really lost here.

Thank you guys.

[lanken]

@kassner, you need to run generator to generate HandlerFactoryDeprecated class.

1. Open terminal.
2. Enter cd <magento_root>/dev/tests/functional/utils.
3. Enter php generate.php.

Currently this class is needed to support deprecated tests which will be rewritten in the future.

[kassner]

Wonderful, I missed the generate.php, now functional tests are working fine, thanks!

Before finishing this issue, back to my first question, bin/magento dev:tests:run should/will be able to run functional tests as well?

[lanken]

Currently we don't an ability to run functional tests via bin/magento dev:tests:run one of the reason it's dependencies to MTF and selenium server, but we will think about it.
Thank you.

[jrosell]

Why not installing MTF and selenium by default?

[mazhalai]

@jrosell please avoid posting on closed issues. It doesn't make sense to install MTF and selenium by default as it will not be used in production mode.