No way to obtain LicenseMatch origin

[srehm]

Description

I was scanning the subversion package from Debian Bullseye (1.14.1-3+deb11u1) with scancode 32.0.8. After the scan completed, I noticed bogus license detections in at least one file: build/ac-macros/swig.m4
Not only are there 51 (if I counted correctly :)) matches from which only the first one is correct. There are also matches in line ranges that don`t even exist in the file (e.g. the file has 360 lines and there are detections on lines 400+).

What makes this even stranger is the fact that in my tests I could not reproduce the error by scanning that file separately. It only happens if I scan the complete source tree. Almost as if the matches are pulled in from other files.

For your convenience I have attached both the sourcecode and the result json.

sourcecode.zip
result.zip

How To Reproduce

• Download scancode v32.0.8 package for python 3.9 from the release page
• Extract and configure
• Download and extract the attached source code
• Change directory to the extracted scancode package
• Run: .\scancode.bat -cli --license-references --license-score 65 --strip-root -n 6 --verbose --json-pp result.json /path/to/extracted/source

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? Windows + Linux
• What version of scancode-toolkit was used to generate the scan file? v32.0.8
• What installation method was used to install/run scancode? Downloaded from release page

[AyanSinhaMahapatra]

@srehm Thanks for reporting! This is a bug indeed.

Almost as if the matches are pulled in from other files.

Yeah, that's the case here actually. 😅

I could not reproduce the error by scanning that file separately.

Yeah in this we could not find the referenced NOTICE file as it was not in the scanned codebase, so we could not add licenses from there.

From the file you referenced see line 2:

See the NOTICE file distributed with this work for additional information

Since there is a reference in this file to the NOTICE file we were getting the license detections from the NOTICE file and adding it back here. But as you've rightly pointed out this is a bit weird and originally we only wanted to do this when it was an unknown reference, but here this is a proper license notice present so we would be doing things a bit differently.
See this issue talked about more detail in #3547 (comment)

[srehm]

Thanks for the explanation. I think I get the point. In my case the swig.m4 references the NOTICE file and that in turn references the LICENSE file. That explains the additional licenses that apply to the file.
However, for the purpose of checking the matches it is very confusing when the lines actually reference a different file. Either an attribute like 'referenced_file' or alternatively remapping start/end line of the match to reflect the reference in the original file (in the case of the swig.m4 that would be lines 3-5) would make things much clearer.
For context, we have a tool that displays the files and visualizes the matches according to the scancode result and currently I dont see how I can filter those referenced matches.

[pombredanne]

@srehm as @AyanSinhaMahapatra pointed, the resolution may be described in #3547

You wrote:

For context, we have a tool that displays the files and visualizes the matches according to the scancode result and currently I dont see how I can filter those referenced matches.

You picked my curiosity. Tell me more!

That said, there are really two issues:

• improving the way we follow references in Refine referenced_filenames in license rules #3547
• tracking the file where a license match originated when we follow references so other tools can leverages this easily, that we can track here.

[AyanSinhaMahapatra]

@pombredanne wrt.

tracking the file where a license match originated when we follow references so other tools can leverages this easily, that we can track here.

Yes! We've discussed this too (to make sure we can distinguish the matches which come from other files in SCIO) and this is somewhere in a branch as I had started working on this. To summarise what we discussed (was to be discussed/reviewed) further to make sure this design is correct:

To pinpoint which file a match is coming from we were going to add a ln attribute somewhat like from_file to the matches which will have two possibile states:

• None: which is the default case where the match is from the present file and not originating from some other file. (Here adding the actual path of the present file would be adding too much info which is not really required)
• path_to_file: when this match originated from some other file and this is the path to that file.

We point to paths and not LicenseDetection id because we carry over all the matches in a file in the following reference case, so this would be enough.

[sschuberth]

To pinpoint which file a match is coming from we were going to add a ln attribute somewhat like from_file to the matches

As the from_file field is available now with ScanCode 32.1.0, can this issue be closed?

[AyanSinhaMahapatra]

Yes, this can be closed, thanks!