Add explicit support for testing: `xmake test`

[paul-reilly]

Is your feature request related to a problem? Please describe.

This is related to target name clashes described in #3380 where multiple subdirectories have a tests target.

Describe the solution you'd like

xmake should have the facility to easily compile and execute tests:

xmake test [target dir or more specific target] [options]

• a new target kind of test would be needed to let xmake know what was a test.

Related to the target clashes described in #3380, when including subdirectories where the author was not aware that defining tests was a going to be a thing in the future, we could have something like :

includes("external/**/xmake.lua", { test_targets = { "tests", "mytests" }})

... which would stop those targets being built as part of a non-test build.

cargo test is highly recommended and since xmake supports languages with integrated testing already:

• xmake test should also be able to run (e.g.) zig test or cargo test for targets using these languages

For complete testing in C/C++:

• support running asan/tsan/ubsan builds

Describe alternatives you've considered

No response

Additional context

No response

[waruqi]

xmake should have the facility to easily compile and execute tests:

why not use xmake run

xmake run target

or run a group of targets (tests)

target("a")
    set_group("test")

target("b")
    set_group("test")

target("c")
    set_group("test")

xmake run -g test

support running asan/tsan/ubsan builds

add_rules("mode.asan", "mode.tsan")

xmake f -m asan
xmake
xnake run -g test

[paul-reilly]

Thanks, I forgot about groups in xmake actually. That's good but wouldn't run cargo test or whatever on targets for other languages.

xmake test could be more universal and cleaner since it would be geared towards testing. If something is a test then:

• naming clashes in includes subdirectories are irrelevant since xmake could build them with temporary names
• all project languages are supported/unified

Re asan,xmake config --mode=asan resets the toolchain at the moment. I switch to clang, change mode to asan and am back on gcc. We have to xmake config --mode=asan --toolchain=clang to preserve the non-default toolchain. Little things like these can be handled automagically.

Anyway, it's just an idea. The mytool test feature is well received in modern language tooling so having it in xmake would be a plus point.

[waruqi]

naming clashes in subdirectories are irrelevant since xmake could build them with temporary namesincludes

xmake test does not solve this problem, it is determined by the loading policy of the xmake.lua interpreter. When xmake loads xmake.lua, all targets with the same name are already merged.

all project languages are supported/unified

xmake run -g test also support them.

I didn't find much difference between xmake run -g test and xmake test.

Re asan, resets the toolchain at the moment. I switch to clang, change mode to asan and am back on gcc. We have to to preserve the non-default toolchain. Little things like these can be handled automagically.xmake config --mode=asanxmake config --mode=asan --toolchain=clang

These are not generic requirements, and xmake test is not very good at wrapping them, they are more like user-specific special requirements that you can solve with a custom task, which is also very handy.

task("test_asan")
    on_run(function ()
        os.exec("xmake f -m asan")
        os.exec("xmake")
        os.exec("xmake run")
    end)

xmake test_asan

[waruqi]

I've considered xmake test before, but so far I haven't found much difference between it and xmake run -g test.

I won't consider it until I know that xmake test can do more than what xmake run -g test can do.

At least what I know now is still not enough.

[becknik]

I'd be also very glad to have a test command. I really think this would lower the redundancy and maybe ease the testing in a time where test-driven development should be the consensus.

I recently discovered this build tool (neat work btw) so I tried to write a template xmake.lua with a test task as you propoed in a commentary above.
f there is a way to do some of these things in neat way in a normal xmake.lua file, I'd be really thankful if you might tell me :)

-- ...
target("<project_name>") -- TODO
	set_default(true)
	set_kind("binary")
	add_files("src/main.cppm", "src/base/*.cppm")

task("test")
	on_run( function()
		os.execute("xmake build -g test -j8")
		os.execute("xmake run -g test")
	end)

target("test_prep")
	set_group("test") -- Ideally should declare a special test group
	set_kind("phony") -- A test kind would be ideal for this
	add_files("src/base/*.cppm")
	-- Test Dependencies:
	add_files("src/base/*.cppm") -- Should include the main projects files to every test-groups target to reduce redundancy
	add_packages("boost/test") -- Should implicitly be included on other test-graoups targets to reduce redundancy

target("blackbox")
	set_group("test")
	set_kind("binary")
	add_files("src/test/blackbox.cppm")

The problem I came across is that there is no way I'm aware of to set "rules" for groups, which might be nice to reduce redundancy. This however could be eased by just adding a set_kind("test") which automatically includes the source files of the same test group/ set_group().

It might be good to have a progress bar for the progress of the test(-groups) execution or messages on test target passes.

I have not mutch experience of this filed of tests, but this are some things I think might be beneficial to this project. Running automated tests properly are features of all big build systems like CTest, gradle, maven etc. .

[jingkaimori]

xmake run -g test will fail at the first non-zero return, thus not all test target are executed. ctest in cmake will run all tests instead. Can xmake has similar behavior with ctest?

[waruqi]

xmake run -g test will fail at the first non-zero return, thus not all test target are executed. ctest in cmake will run all tests instead. Can xmake has similar behavior with ctest?

No way now, you can print error logs directly and return zero code in test target.

[waruqi]

I have supported it now. #4259

$ xmake update -s dev

Add test to target (no arguments)

target("test")
    add_tests("testname")

Add run arguments

target("test")
    add_tests("testname", {runargs = "arg1"})
    add_tests("testname", {runargs = {"arg1", "arg2"}})

It will use global target/set_runargs if no {runargs = } in add_tests

target("test")
    add_tests("testname")
    set_runargs("arg1", "arg2")

Set run directory

target("test")
    add_tests("testname", {rundir = os.projectdir()})

It will use global target/set_rundir if no {rundir = } in add_tests

target("test")
    add_tests("testname")
    set_rundir("$(projectdir)")

Set run environments

target("test")
    add_tests("testname", {runenvs = {LD_LIBRARY_PATH = "/lib"}})

It will use global target/add_runenvs if no {runenvs = } in add_tests

target("test")
    add_tests("testname")
    add_runenvs("LD_LIBRARY_PATH", "/lib")

Match passed outputs

It also supports lua pattern matching.

target("test")
    add_tests("testname", {pass_outputs = "hello"})
    add_tests("testname", {pass_outputs = "hello *"})
    add_tests("testname", {pass_outputs = {"hello", "hello *"}})

Match failed outputs

It also supports lua pattern matching.

target("test")
    add_tests("testname", {fail_outputs = "hello"})
    add_tests("testname", {fail_outputs = "hello *"})
    add_tests("testname", {fail_outputs = {"hello", "hello *"}})

Trim output first before matching pass/fail outputs

target("test")
    add_tests("testname", {trim_output = true, pass_outputs = "foo", fail_outputs = "hello"})

Plain match output

we can also set {plain = true} to force to use plain text matching.

It will disable lua pattern matching.

target("test")
    add_tests("testname", {plain = true, pass_outputs = "foo", fail_outputs = "hello"})

Set test group

target("test")
    add_tests("testname", {group = "foo"})

It will use global target/set_group if no {group = } in add_tests

target("test")
    add_tests("testname")
    set_group("foo")

then we can run the tests with the given group name, and it also supports lua pattern matching.

$ xmake test -g "foo"
$ xmake test -g "foo*"

Run the given tests

we need pass target and test names, e.g.

$ xmake test targetname/testname

And we can pass lua pattern to match some tests

$ xmake test */testname
$ xmake test targetname/*
$ xmake test targetname/foo*

Automatically build

It will be built automatically even if we set default = false

target("test")
    add_tests("testname")
    set_default(false)

$ xmake test
[ 25%]: cache compiling.release src/main.cpp
[ 50%]: linking.release test
running tests ...
[100%]: test/testname .................................... passed 6.000s

100% tests passed, 0 tests failed out of 1, spent 0.006s

Custom scripts

We can define before_test, on_test and after_test to custom test script in rule/target.

target("test")
     on_test(function (target, opt)
        print(opt.name, opt.runenvs, opt.runargs, opt.pass_outputs)

        -- do test
        -- ...

        -- passed 
        return true
       
        -- failied
        return false, errors
     end)

Parallel test

$ xmake test -jN

Examples

add_rules("mode.debug", "mode.release")

for _, file in ipairs(os.files("src/test_*.cpp")) do
    local name = path.basename(file)
    target(name)
        set_kind("binary")
        set_default(false)
        add_files("src/" .. name .. ".cpp")
        add_tests("default")
        add_tests("args", {runargs = {"foo", "bar"}})
        add_tests("pass_output", {trim_output = true, runargs = "foo", pass_outputs = "hello foo"})
        add_tests("fail_output", {fail_outputs = {"hello2 .*", "hello xmake"}})
end

ruki-2:test ruki$ xmake test
running tests ...
[  2%]: test_1/args        .................................... passed 7.000s
[  5%]: test_1/default     .................................... passed 5.000s
[  8%]: test_1/fail_output .................................... passed 5.000s
[ 11%]: test_1/pass_output .................................... passed 6.000s
[ 13%]: test_2/args        .................................... passed 7.000s
[ 16%]: test_2/default     .................................... passed 6.000s
[ 19%]: test_2/fail_output .................................... passed 6.000s
[ 22%]: test_2/pass_output .................................... passed 6.000s
[ 25%]: test_3/args        .................................... passed 7.000s
[ 27%]: test_3/default     .................................... passed 7.000s
[ 30%]: test_3/fail_output .................................... passed 6.000s
[ 33%]: test_3/pass_output .................................... passed 6.000s
[ 36%]: test_4/args        .................................... passed 6.000s
[ 38%]: test_4/default     .................................... passed 6.000s
[ 41%]: test_4/fail_output .................................... passed 5.000s
[ 44%]: test_4/pass_output .................................... passed 6.000s
[ 47%]: test_5/args        .................................... passed 5.000s
[ 50%]: test_5/default     .................................... passed 6.000s
[ 52%]: test_5/fail_output .................................... failed 6.000s
[ 55%]: test_5/pass_output .................................... failed 5.000s
[ 58%]: test_6/args        .................................... passed 7.000s
[ 61%]: test_6/default     .................................... passed 6.000s
[ 63%]: test_6/fail_output .................................... passed 6.000s
[ 66%]: test_6/pass_output .................................... passed 6.000s
[ 69%]: test_7/args        .................................... failed 6.000s
[ 72%]: test_7/default     .................................... failed 7.000s
[ 75%]: test_7/fail_output .................................... failed 6.000s
[ 77%]: test_7/pass_output .................................... failed 5.000s
[ 80%]: test_8/args        .................................... passed 7.000s
[ 83%]: test_8/default     .................................... passed 6.000s
[ 86%]: test_8/fail_output .................................... passed 6.000s
[ 88%]: test_8/pass_output .................................... failed 5.000s
[ 91%]: test_9/args        .................................... passed 6.000s
[ 94%]: test_9/default     .................................... passed 6.000s
[ 97%]: test_9/fail_output .................................... passed 6.000s
[100%]: test_9/pass_output .................................... passed 6.000s

80% tests passed, 7 tests failed out of 36, spent 0.242s

Verbose/Diagnosis output

$ xmake test -vD

[waruqi]

Return exit code (-1) on failure (default)

It will signal a test failure to CI or other tools. If we can suppress it, we need set

set_policy("test.return_zero_on_failure", true)

Stop to test on the first failure

set_policy("test.stop_on_first_failure", true)