Missing source-based coverage for a source file which is supposed to be covered

[marco-c]

As I mentioned in rust-lang/compiler-team#278 (comment), I've implemented support for source-based code coverage in grcov, and started using it to collect coverage for grcov itself.

There is a file that is definitely covered, but doesn't appear to be in the report. It is in the report when using gcov-based coverage.

STR:

1. Clone https://github.com/mozilla/grcov;
2. Run LLVM_PROFILE_FILE="grcov-%p-%m.profraw" cargo test;
3. Run grcov `find . \( -name "grcov-*.profraw" \) -print\` --binary-path ./target/debug/grcov -s . -t html --branch --ignore-not-existing --ignore "/*" -o coverage to generate a HTML report (or you can also generate a textual one using the llvm tools);
4. See that src/llvm_tools.rs appears to be totally uncovered, while it should be covered.

[nagisa]

Does this properly associate the profraw with the exact binary that produced it? --binary-path ./target/debug/grcov would suggest that you're using grcov (the [[bin]] target) while the profraw files seem to have been generated by various test binaries (typically have hashes in them, somewhere).

Is it also reproducible when using the llvm-cov tooling?

[marco-c]

Does this properly associate the profraw with the exact binary that produced it? --binary-path ./target/debug/grcov would suggest that you're using grcov (the [[bin]] target) while the profraw files seem to have been generated by various test binaries (typically have hashes in them, somewhere).

Oh, you're right, I was assuming only the source code behind #[cfg(test)] would be missed this way. It works if I use the actual test binary!
This means grcov will need a way to automatically detect the binary paths, and run llvm-cov accordingly...

[nagisa]

Closing as not-a-bug (at least not in rustc).

Of course, I agree that having to associate the exact binary with the profile file it produced is sub-optimal, and cargo makes it harder still by appending hashes to everything. That said, this seems like the issue to be fixed in either the LLVM-provided instrumentation runtime (maybe just having a way for it to splice binary name into the profile filename would help?) or in cargo (that's what I'm planning to do for my own use-cases).

[marco-c]

That said, this seems like the issue to be fixed in either the LLVM-provided instrumentation runtime (maybe just having a way for it to splice binary name into the profile filename would help?) or in cargo (that's what I'm planning to do for my own use-cases).

Maybe using %m in LLVM_PROFILE_FILE, which "expands out to the instrumented binary’s signature". Then we just need to read the signature from the binaries to match them to the right profraw file.

[richkadel]

@marco-c - In case this is helpful...

When PR #79109 lands, there will be an update to the "Rust Unstable Book" section on source-based-code-coverage, including an example for coverage of tests run through cargo test.

The example does use %m to make sure each test gets its own profraw result.

You can see the new version in the PR commit, here:

https://github.com/rust-lang/rust/blob/00117d712189ca703b481b471e11bcc742f0346e/src/doc/unstable-book/src/compiler-flags/source-based-code-coverage.md#test-coverage

Hope this helps.

[marco-c]

@richkadel for now in grcov I'm simply running llvm-cov with the profdata file associated to all possible binary files, not a great solution, but it seems to work...

The next step will be to figure out how to use the binary signature to correctly match the profraw to the right binary, but I've looked at the binary signature code in LLVM and it is platform-specific, I'm not too keen on including something like that in grcov.
I've just now realized the binary signature is in the binary name itself! So that's going to make it easier ;)
The only constraint on the user will be the forced usage of LLVM_PROFILE_FILE.

For context, grcov is a tool to parse coverage artifacts (from Rust and other languages). Basically an alternative to llvm-cov, but easier to use and supporting more options, input formats, and output formats. I need to make it easy to use as part of CI, so can't rely on users running cargo test, checking the name of the binaries, and passing their names to grcov.

This is what I currently suggest doing to people using grcov:

LLVM_PROFILE_FILE="your_name-%p-%m.profraw" cargo test --verbose
grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "/*" -o lcov.info