test: these may need to be fixed/re-enabled in future

[terriko]

=========================== short test summary info ============================
FAILED test/test_csv2cve.py::TestCSV2CVE::test_csv2cve_valid_file - AssertionError: assert ('cve_bin_tool', 20, 'There are 2 products with known CVEs detected') in [('cve_bin_tool', 20, 'CVE Binary Tool v3.3.1dev0'), ('cve_bin_tool', 20, 'This product uses the NVD API but is not endorsed or certified by the NVD.'), ('cve_bin_tool.CVEDB', 20, 'Using cached CVE data (<24h old). Use -u now to update immediately.'), ('cve_bin_tool.CVEDB', 20, 'There are 151760 CVE entries in the database'), ('cve_bin_tool.CVEDB', 20, 'There are 133779 CVE entries from OSV in the database'), ('cve_bin_tool.CVEDB', 20, 'There are 17981 CVE entries from REDHAT in the database'), ...]
 +  where [('cve_bin_tool', 20, 'CVE Binary Tool v3.3.1dev0'), ('cve_bin_tool', 20, 'This product uses the NVD API but is not endorsed or certified by the NVD.'), ('cve_bin_tool.CVEDB', 20, 'Using cached CVE data (<24h old). Use -u now to update immediately.'), ('cve_bin_tool.CVEDB', 20, 'There are 151760 CVE entries in the database'), ('cve_bin_tool.CVEDB', 20, 'There are 133779 CVE entries from OSV in the database'), ('cve_bin_tool.CVEDB', 20, 'There are 17981 CVE entries from REDHAT in the database'), ...] = <_pytest.logging.LogCaptureFixture object at 0x7fa0d05c4d30>.record_tuples
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[True-exploits_list0-product_info0-triage_info0-CRITICAL-EXPLOIT] - IndexError: list index out of range
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[False-exploits_list1-product_info1-triage_info1-CRITICAL] - IndexError: list index out of range
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[True-exploits_list2-product_info2-triage_info2-CRITICAL] - IndexError: list index out of range
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[False-exploits_list3-product_info3-triage_info3-CRITICAL] - IndexError: list index out of range
FAILED test/test_language_scanner.py::TestLanguageScanner::test_python_package[/home/runner/work/cve-bin-tool/cve-bin-tool/test/language_data/PKG-INFO] - AssertionError: assert ProductInfo(v...O', purl=None) == ProductInfo(v...O', purl=None)
  
  Omitting 4 identical items, use -vv to show
  Differing attributes:
  ['vendor']
  
  Drill down into differing attribute vendor:
    vendor: 'UNKNOWN' != 'facebook'
    - facebook
    + UNKNOWN
  
  Full diff:
  - ProductInfo(vendor='facebook', product='zstandard', version='0.18.0', location='/home/runner/work/cve-bin-tool/cve-bin-tool/test/language_data/PKG-INFO', purl=None)
  ?                     ^^^^^^^^
  + ProductInfo(vendor='UNKNOWN', product='zstandard', version='0.18.0', location='/home/runner/work/cve-bin-tool/cve-bin-tool/test/language_data/PKG-INFO', purl=None)
  ?                     ^^^^^^^
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx-spdx_parsed_data0] - AssertionError: assert ProductInfo(vendor='saxon', product='saxon', version='8.8', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.rdf-spdx_parsed_data1] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='unknown', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.json-spdx_parsed_data2] - AssertionError: assert ProductInfo(vendor='saxon', product='saxon', version='8.8', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.xml-spdx_parsed_data3] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='unknown', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.yml-spdx_parsed_data4] - AssertionError: assert ProductInfo(vendor='saxon', product='saxon', version='8.8', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.yaml-spdx_parsed_data5] - AssertionError: assert ProductInfo(vendor='saxon', product='saxon', version='8.8', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_mixed_test.spdx.json-spdx_parsed_data6] - AssertionError: assert ProductInfo(vendor='saxon', product='saxon', version='8.8', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_test.xml-cyclonedx_parsed_data0] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='acme', product='application', version='9.1.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='unknown', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_test.json-cyclonedx_parsed_data1] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='acme', product='application', version='9.1.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='unknown', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_test2.json-cyclonedx_parsed_data2] - AssertionError: assert ProductInfo(vendor='ubuntu', product='ubuntu', version='22.04', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='acme', product='application', version='9.1.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='web-framework', version='1.0.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='ubuntu', version='22.04', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='boot-prom', version='1.0.9', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_mixed_test.json-cyclonedx_parsed_data3] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1', location='NotFound', purl=None) in defaultdict(<class 'dict'>, {ProductInfo(vendor='acme', product='application', version='9.1.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='unknown', product='glibc', version='2.11.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='invalid_purl_package', version='1.1.0', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_bad_ext_ref_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_bad_cpe22.json-cyclonedx_parsed_data0] - AssertionError: assert ProductInfo(vendor='libexpat_project', product='libexpat', version='2.0.1', location='NotFound', purl=None) in dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None)])
 +  where dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None)]) = <built-in method keys of collections.defaultdict object at 0x7f6dcccb44f0>()
 +    where <built-in method keys of collections.defaultdict object at 0x7f6dcccb44f0> = defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}}).keys
FAILED test/test_sbom.py::TestSBOM::test_bad_ext_ref_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_bad_cpe23.json-cyclonedx_parsed_data1] - AssertionError: assert ProductInfo(vendor='libexpat_project', product='libexpat', version='2.0.1', location='NotFound', purl=None) in dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None)])
 +  where dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None)]) = <built-in method keys of collections.defaultdict object at 0x7f6dccb882c0>()
 +    where <built-in method keys of collections.defaultdict object at 0x7f6dccb882c0> = defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}}).keys
FAILED test/test_sbom.py::TestSBOM::test_bad_ext_ref_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_bad_purl.json-cyclonedx_parsed_data2] - AssertionError: assert ProductInfo(vendor='libexpat_project', product='libexpat', version='2.0.1', location='NotFound', purl=None) in dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None)])
 +  where dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None)]) = <built-in method keys of collections.defaultdict object at 0x7f6dd28c4130>()
 +    where <built-in method keys of collections.defaultdict object at 0x7f6dd28c4130> = defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='libjpeg-novendor', version='8b', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='libexpat', version='2.0.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='ncurses-noversion', version='5.9.noversion', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}}).keys
FAILED test/test_sbom.py::TestSBOM::test_ext_ref_priority_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_ext_ref_priority.json-cyclonedx_parsed_data0] - AssertionError: assert ProductInfo(vendor='gnu', product='ncurses', version='5.9', location='NotFound', purl=None) in dict_keys([ProductInfo(vendor='ijg', product='libjpeg', version='8b', location='NotFound', purl=None), ProductInfo(vendor='libexpat_project', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='zlib', version='1.2.3', location='NotFound', purl=None)])
 +  where dict_keys([ProductInfo(vendor='ijg', product='libjpeg', version='8b', location='NotFound', purl=None), ProductInfo(vendor='libexpat_project', product='libexpat', version='2.0.1', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='zlib', version='1.2.3', location='NotFound', purl=None)]) = <built-in method keys of collections.defaultdict object at 0x7f6dccd5a160>()
 +    where <built-in method keys of collections.defaultdict object at 0x7f6dccd5a160> = defaultdict(<class 'dict'>, {ProductInfo(vendor='ijg', product='libjpeg', version='8b', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='libexpat_project', product='libexpat', version='2.0.1', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='unknown', product='ncurses', version='5.9', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='unknown', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}}).keys
FAILED test/test_sbom.py::TestSBOM::test_common_prefix_split[openzeppelin-contracts-4.8.1-productinfo0-no_existent_file] - AssertionError: assert ProductInfo(vendor='openzeppelin', product='contracts', version='4.8.1', location='NotFound', purl=None) in []
FAILED test/test_sbom.py::TestSBOM::test_common_prefix_split[rubygem-builder-3.2.4-productinfo1-no_existent_file] - AssertionError: assert ProductInfo(vendor='downline_goldmine', product='builder', version='3.2.4', location='NotFound', purl=None) in []
FAILED test/test_triage.py::TestTriage::test_json - assert 0 >= 1
 +  where 0 = len([])
==== 24 failed, 1988 passed, 21 skipped, 41 warnings in 1433.83s (0:23:53) =====

This looks like one of those days where vendors wind up marked as UNKNOWN, which I thought was fixed by the recent changes to the OSV parsing of linked issues. So I'm not sure what's going on here.

@mastersans and @inosmeet -- if these are blocking you from continuing to work / get PRs merged, please go ahead and make a PR that moves those 24 tests into the network-mayfail github actions job and we'll sort them out later.

[mastersans]

@terriko @inosmeet i re-ran the test and seems they are passing again.

[inosmeet]

yeah! might be due to bad cache

[mastersans]

and they are failing againn........ : (

[terriko]

Updating title/milestone. These have been disabled but we'll probably want to re-enable them or replace them with something more stable circa the next release.

[terriko]

@inosmeet

Does test/test_language_scanner.py::TestLanguageScanner::test_python_package need to be replaced so it's looking for something that isn't zstandard? We could probably switch in a pkg-info file from reportlab for that test if it makes it work better for us in future.

[terriko]

Did a quick test on these and at least some of them are failing:

• test: re-enable disabled tests #4353

It looked like the first failure was related to triage and might have been replaced by the new tests anyhow. I'm going to let it run and see where we're at, but I may push this out from 3.4 into future so I can do a better job with fixing it.

[mastersans]

Did a quick test on these and at least some of them are failing:

• test: re-enable disabled tests #4353

It looked like the first failure was related to triage and might have been replaced by the new tests anyhow. I'm going to let it run and see where we're at, but I may push this out from 3.4 into future so I can do a better job with fixing it.

I'll try to get it fixed asap but i guess you are right that those are now replace with newer tests.

[terriko]

4 failures from my PR. The python language package one is my fault (I was trying to change the check so looked for zstandard but not facebook as a vendor) and should be fixable. haven't looked at the rest yet. Two triage ones and one sbom one with the unknown problem. I'm currently thinking that I should (temporarily) change the sbom test to not check vendor so that we don't have to keep disabling it, though obviously I'd rather fix our race condition more permanently.

 =========================== short test summary info ============================
FAILED test/test_triage.py::TestTriage::test_json - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/requirements_scan-4fz1vrlt/test_triage_output.json'
FAILED test/test_triage.py::TestTriage::test_vex - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/requirements_scan-4fz1vrlt/test_triage_output.json'
FAILED test/test_language_scanner.py::TestLanguageScanner::test_python_package[/opt/actions-runner/_work/cve-bin-tool/cve-bin-tool/test/language_data/PKG-INFO] - TypeError: tuple indices must be integers or slices, not str
FAILED test/test_sbom.py::TestSBOM::test_ext_ref_priority_cyclonedx_file[/opt/actions-runner/_work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_ext_ref_priority.json-cyclonedx_parsed_data0] - AssertionError: assert ProductInfo(vendor='ijg', product='libjpeg', version='8b', location='NotFound', purl=None) in dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-3', version='8b-2', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat-3', version='2.0.1-3', location='NotFound', purl=None), ProductInfo(vendor='gnu', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='ncurses_project', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None), ProductInfo(vendor='gnu', product='zlib', version='1.2.3', location='NotFound', purl=None), ProductInfo(vendor='cloudflare', product='zlib', version='1.2.3', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='zlib', version='1.2.3', location='NotFound', purl=None)])
 +  where dict_keys([ProductInfo(vendor='UNKNOWN', product='libjpeg-3', version='8b-2', location='NotFound', purl=None), ProductInfo(vendor='UNKNOWN', product='libexpat-3', version='2.0.1-3', location='NotFound', purl=None), ProductInfo(vendor='gnu', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='ncurses_project', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='ncurses', version='5.9', location='NotFound', purl=None), ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None), ProductInfo(vendor='gnu', product='zlib', version='1.2.3', location='NotFound', purl=None), ProductInfo(vendor='cloudflare', product='zlib', version='1.2.3', location='NotFound', purl=None), ProductInfo(vendor='unknown', product='zlib', version='1.2.3', location='NotFound', purl=None)]) = <built-in method keys of collections.defaultdict object at 0x7fe28479e9d0>()
 +    where <built-in method keys of collections.defaultdict object at 0x7fe28479e9d0> = defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='libjpeg-3', version='8b-2', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='libexpat-3', version='2.0.1-3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='gnu', product='ncurses', version='5.9', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='ncurses_project', product='ncurses', version='5.9', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='unknown', product='ncurses', version='5.9', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='zlib', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='gnu', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='cloudflare', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='unknown', product='zlib', version='1.2.3', location='NotFound', purl=None): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}}).keys
===== 4 failed, 273 passed, 1754 skipped, 40 warnings in 911.37s (0:15:11) =====

[inosmeet]

@inosmeet

Does test/test_language_scanner.py::TestLanguageScanner::test_python_package need to be replaced so it's looking for something that isn't zstandard? We could probably switch in a pkg-info file from reportlab for that test if it makes it work better for us in future.

If the file from reportlab contains more products, we can definitely switch. Testing with more data will be helpful.

[terriko]

@inosmeet I think pkg-info generally has a single product in it, but reportlab has the (dubious) benefit of having a cve that is valid and will continue to be reported which is convenient for testing purposes.

[terriko]

Considering changes to make these more robust:

• we could really use a helper function that compared big data structures section-by-section so that the error messages (and tuning) could be easier
• it seems like a lot of these would work ok even in the face of our specific problem if we just didn't assert against vendor
• i'd really rather fix things so we don't have the kind of race condition that we've been hitting, though. There's definitely still something up with OSV loading that I can't explain.

I'm going to go back to the PR and see what things we can re-enable "easily" then probably punt the rest out to the future milestone so they can be solved more completely.