Add Package Instances #2691 #2825

AyanSinhaMahapatra · 2022-01-31T10:17:05Z

This PR adds the PackageInstance class and functions to group package
manifests and package data as top level package instances.

Existing package data are ported to this new approach.

Reference: #2098
Reference: #2691
Reference: #2692
Reference: #2693
Reference: #2843
Reference: #2652
Signed-off-by: Ayan Sinha Mahapatra [email protected]

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁

Adds the PackageInstance class and functions to group package manifests and create a package instance out of them. Adds a sample example case for python. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds functions to merge package data from multiple package manifests into a package instance. Adds tests for a simple python manifests case. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Creates PackageManifest classes for cran description manifests and overrides the methods for detection and PackageManifest creation. See #2748 Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Creates PackageManifest class for maven pom.xml manifests and overrides the methods for detection and PackageManifest creation. See #2748 Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

JonoYang

@AyanSinhaMahapatra Initial work on the PackageInstance model looks good. I just have some minor things I pointed out.

tests/packagedcode/test_package_instance.py

src/packagedcode/__init__.py

src/packagedcode/models.py

Modifies package instance tests to remove package_uuid fields, and adds tests for package instance/manifest type uniqueness. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds package_uuid as the pURL string for the package instance, with it's UUID as a qualifier in the pURL. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

pombredanne · 2022-02-07T09:16:35Z

re: #2825 (comment)

I was having some problems with the package_uuid field as it had a UUID that was different for each test run. So I forgot to make this False, now this is resolved. I've also modified the check_json_scan to accept a new argument remove_package_uuid which removes the UUID qualifier from the package_uuid pURL string.

There are three ways to handle these cases:

ignore it as you have done but you need to have at least one test case that would not ignore the UUID to ensure this is generated correctly.
use mocks in tests so that you have a stable mocked UUID created and do not need to ignore it
add something like an extra _test=True argument to the tested function(s) that is called only in tests and that creates a static UUID that is well known for testing.

The more generic solution is 2. ... use mock.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds `for_packages` attribute to files to have `package_uuid` fields for the package instance this file belongs to. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Modify and rename the resource-level attribute added for a package scan to "package_data", replacing "package_manifests". Also replaces "package_manifest_paths" in top-level "packages" to "package_data_paths". Changes API function get_package_manifests to get_package_data. Also renames other variables with similar naming. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Regenate tests after "package_manifests" -> "package_data" renaming and add the corresponding chages to the test files. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds DependencyInstance to packagedcode models. Adds top-level dependencies list for DependencyInstance objects. Adds DependencyInstance creation to process_codebase in packagedcode plugin. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Regenerates test expectations with top-level `dependencies` list. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Modify `remove_package_uuid` flag to `remove_instance_uuid` to include dependencies. Add `ignore_instance_uuid` attribute and a new test that has UUIDs in the test file. Add XFAIL for failing test. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add node_modules as ignored paths for NPM package instance creation. Regenerate test expectations after merging from develop. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds fix for nuget and maven bugs which were in the way of activating their respective package instance creation. Also fixes URL creation for lockfiles. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Package checksums were being wrongly ignored in package instance data update from detected package data, this commit fixes that. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Update Package->PackageData, PackageInstance->Package and PackageData->PackageDataFile for consistency. Also rename "package_data_paths" to "package_data_files" and "is_package_data" to "is_package_data_file". Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This updates the CHANGELOG with details about specific changes in package detection, in the API and the data model. Also bumps the scancode data format to 3.0.0 as there are major data format changes. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Rename "requirement" to "extracted_requirement" and add new attribute "resolved_package". Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra · 2022-03-04T21:16:55Z

@pombredanne This is ready!

pombredanne

LGTM. Thanks! merging now.

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 37434fc to 4665b2d Compare February 1, 2022 14:00

AyanSinhaMahapatra added 4 commits February 1, 2022 21:13

Add initial package instance creation

c889023

Adds the PackageInstance class and functions to group package manifests and create a package instance out of them. Adds a sample example case for python. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add function to merge package data

fa010e4

Adds functions to merge package data from multiple package manifests into a package instance. Adds tests for a simple python manifests case. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds PackageManifest classes for cran

9f9e3a6

Creates PackageManifest classes for cran description manifests and overrides the methods for detection and PackageManifest creation. See #2748 Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds PackageManifest class for maven

695ae3b

Creates PackageManifest class for maven pom.xml manifests and overrides the methods for detection and PackageManifest creation. See #2748 Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 4665b2d to 695ae3b Compare February 1, 2022 15:48

pombredanne mentioned this pull request Feb 2, 2022

Handle deduplication of packages in a scan #2281

Closed

JonoYang reviewed Feb 2, 2022

View reviewed changes

tests/packagedcode/test_package_instance.py Outdated Show resolved Hide resolved

src/packagedcode/__init__.py Outdated Show resolved Hide resolved

src/packagedcode/__init__.py Show resolved Hide resolved

src/packagedcode/models.py Show resolved Hide resolved

AyanSinhaMahapatra added 2 commits February 6, 2022 22:36

Modify package instance tests

0ef84fc

Modifies package instance tests to remove package_uuid fields, and adds tests for package instance/manifest type uniqueness. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add proper package_uuid

88257a5

Adds package_uuid as the pURL string for the package instance, with it's UUID as a qualifier in the pURL. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra mentioned this pull request Feb 7, 2022

Add PackageManifest Class #2748

Merged

4 tasks

AyanSinhaMahapatra added 7 commits February 14, 2022 14:04

Update tests and expectations for python package instance

2ca018b

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Activate package instance creation for others

89b8302

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Update test expectations for top level packages

1301612

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Remove obsolete root_path

c4c6006

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Fix package instance creation bugs

35529b7

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Update test expectaions for package instances

3c86b7c

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add for_packages file level attribute #2693

3f93f5e

Adds `for_packages` attribute to files to have `package_uuid` fields for the package instance this file belongs to. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 8768acc to ddcd536 Compare February 16, 2022 13:44

Update test expectations with for_package #2693

beeef5a

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from ddcd536 to beeef5a Compare February 16, 2022 14:15

AyanSinhaMahapatra added 7 commits February 18, 2022 22:56

Add changes in test data for "package_data" renaming #2843

83c6588

Regenate tests after "package_manifests" -> "package_data" renaming and add the corresponding chages to the test files. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add top-level dependencies list #2843

00d4f71

Adds DependencyInstance to packagedcode models. Adds top-level dependencies list for DependencyInstance objects. Adds DependencyInstance creation to process_codebase in packagedcode plugin. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Modifies test expectations #2843

abc807e

Regenerates test expectations with top-level `dependencies` list. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Merge branch 'develop' into 2098-top-level-packages

498080b

Fix test failures from merging develop

7497407

Add node_modules as ignored paths for NPM package instance creation. Regenerate test expectations after merging from develop. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 27ca9ac to 7497407 Compare February 23, 2022 15:11

Fix maven and nuget bugs

0f57bce

Adds fix for nuget and maven bugs which were in the way of activating their respective package instance creation. Also fixes URL creation for lockfiles. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 2366754 to 32c160b Compare March 3, 2022 13:16

AyanSinhaMahapatra added 2 commits March 3, 2022 20:05

Fix package checksum reporting

9688dfc

Package checksums were being wrongly ignored in package instance data update from detected package data, this commit fixes that. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Fix dparse2 pypi detection bug

b6bb763

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 32c160b to b6bb763 Compare March 3, 2022 14:37

AyanSinhaMahapatra added 4 commits March 4, 2022 19:24

Remove redundant dependency info from packages

f2f998f

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Merge branch 'develop' into 2098-top-level-packages

facad19

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Update build_gradle.py to new model names

527e1dd

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra requested a review from pombredanne March 4, 2022 17:18

AyanSinhaMahapatra added 4 commits March 4, 2022 22:52

Fix type/class ambiguity in package class lists

8abfe18

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Remove "root_path" package data attribute

c0475ea

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Modify DependentPackage Atrributes

376abc6

Rename "requirement" to "extracted_requirement" and add new attribute "resolved_package". Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 7c95531 to 376abc6 Compare March 4, 2022 21:16

pombredanne approved these changes Mar 5, 2022

View reviewed changes

pombredanne merged commit e080f83 into develop Mar 5, 2022

pombredanne deleted the 2098-top-level-packages branch March 5, 2022 05:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add Package Instances #2691 #2825

Add Package Instances #2691 #2825

Uh oh!

AyanSinhaMahapatra commented Jan 31, 2022 •

edited by pombredanne

Loading

Uh oh!

JonoYang left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne commented Feb 7, 2022

Uh oh!

AyanSinhaMahapatra commented Mar 4, 2022

Uh oh!

pombredanne left a comment

Uh oh!

Uh oh!

Uh oh!

Add Package Instances #2691 #2825

Add Package Instances #2691 #2825

Uh oh!

Conversation

AyanSinhaMahapatra commented Jan 31, 2022 • edited by pombredanne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

JonoYang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne commented Feb 7, 2022

Uh oh!

AyanSinhaMahapatra commented Mar 4, 2022

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AyanSinhaMahapatra commented Jan 31, 2022 •

edited by pombredanne

Loading