Enhancements in EC2 and Added Support for AMIs #633

kedarghule · 2021-06-01T14:52:35Z

This PR does the following:

Currently the auto_scaling_groups.py code does not use the UNWIND approach, instead it uses loops to populate the AWS resources. This PR rectifies that. Additionally, some more attributes of AutoScalingGroups are populated.
Some additional useful attributes missing from EC2 Instances and Subnets get populated too! Corresponding changes have been made in the documentation. (I specifically didn't add the UNWIND approach here because I wasn't quite sure of the code structure here and I didn't want to break things 🙈)
The documentation for Auto Scaling Groups was missing all this while. Added it!
And the one major update in this PR is - Added support for Amazon Machine Images (AMIs) - images.py in the ec2 folder.

# Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.

…artography into aws-ec2-enhancements

achantavy

Left a couple of questions but it's looking really good! Thanks for taking this on.

achantavy · 2021-06-10T06:09:29Z

docs/schema/aws.md

+
+## EC2Image
+
+Representation of an AWS [EC2 Images (AMIs)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html).


Let's also link to the API doc: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Image.html

achantavy · 2021-06-10T06:10:46Z

docs/schema/aws.md

+
+## AutoScalingGroup
+
+Representation of an AWS [Auto Scaling Group Resource](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html).


Thanks for including this. Can you also add a link to the AWS API doc?

achantavy · 2021-06-10T06:16:05Z

cartography/intel/aws/ec2/images.py

+) -> None:
+    ingest_images = """
+    UNWIND {images_list} as image
+        MERGE (i:EC2Image{id: image.ImageId})


Is the ImageId guaranteed to be unique across all AWS like an ARN?

If ImageId is only unique per region, let's set i.id to be {ImageId}|{region}, set i.imageid = {ImageId}, and also add an index for i.imageid.

cc: @mpurusottamc

@achantavy AMI ID are unique per region. So, we will follow your suggestion of appending region to the id.
Finding an AMI
Thanks @kedarghule

@achantavy and @mpurusottamc Changes made and committed 🚀

achantavy · 2021-06-10T06:22:24Z

cartography/intel/aws/ec2/auto_scaling_groups.py

    """

+    # neo4j does not accept datetime objects and values. This loop is used to convert


Nit: technically not correct anymore as of 3.5 iirc, but it's fine to keep as str().

Ah I see. Okay, I'll just remove that comment.

kedarghule · 2021-06-15T07:03:03Z

@achantavy All suggested changes as per your review are done here. Anymore changes required? :)

achantavy

Very much appreciate the contribution as always @kedarghule :D

Bumps [pytest-cov](https://github.com/pytest-dev/pytest-cov) from 2.10.0 to 6.0.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest-cov/blob/master/CHANGELOG.rst">pytest-cov's changelog</a>.</em></p> <blockquote> <h2>6.0.0 (2024-10-29)</h2> <ul> <li>Updated various documentation inaccuracies, especially on subprocess handling.</li> <li>Changed fail under checks to use the precision set in the coverage configuration. Now it will perform the check just like <code>coverage report</code> would.</li> <li>Added a <code>--cov-precision</code> cli option that can override the value set in your coverage configuration.</li> <li>Dropped support for now EOL Python 3.8.</li> </ul> <h2>5.0.0 (2024-03-24)</h2> <ul> <li>Removed support for xdist rsync (now deprecated). Contributed by Matthias Reichenbach in <code>[#623](pytest-dev/pytest-cov#623) <https://github.com/pytest-dev/pytest-cov/pull/623></code>_.</li> <li>Switched docs theme to Furo.</li> <li>Various legacy Python cleanup and CI improvements. Contributed by Christian Clauss and Hugo van Kemenade in <code>[#630](pytest-dev/pytest-cov#630) <https://github.com/pytest-dev/pytest-cov/pull/630></code><em>, <code>[#631](pytest-dev/pytest-cov#631) <https://github.com/pytest-dev/pytest-cov/pull/631></code></em>, <code>[#632](pytest-dev/pytest-cov#632) <https://github.com/pytest-dev/pytest-cov/pull/632></code>_ and <code>[#633](pytest-dev/pytest-cov#633) <https://github.com/pytest-dev/pytest-cov/pull/633></code>_.</li> <li>Added a <code>pyproject.toml</code> example in the docs. Contributed by Dawn James in <code>[#626](pytest-dev/pytest-cov#626) <https://github.com/pytest-dev/pytest-cov/pull/626></code>_.</li> <li>Modernized project's pre-commit hooks to use ruff. Initial POC contributed by Christian Clauss in <code>[#584](pytest-dev/pytest-cov#584) <https://github.com/pytest-dev/pytest-cov/pull/584></code>_.</li> <li>Dropped support for Python 3.7.</li> </ul> <h2>4.1.0 (2023-05-24)</h2> <ul> <li>Updated CI with new Pythons and dependencies.</li> <li>Removed rsyncdir support. This makes pytest-cov compatible with xdist 3.0. Contributed by Sorin Sbarnea in <code>[#558](pytest-dev/pytest-cov#558) <https://github.com/pytest-dev/pytest-cov/pull/558></code>_.</li> <li>Optimized summary generation to not be performed if no reporting is active (for example, when <code>--cov-report=''</code> is used without <code>--cov-fail-under</code>). Contributed by Jonathan Stewmon in <code>[#589](pytest-dev/pytest-cov#589) <https://github.com/pytest-dev/pytest-cov/pull/589></code>_.</li> <li>Added support for JSON reporting. Contributed by Matthew Gamble in <code>[#582](pytest-dev/pytest-cov#582) <https://github.com/pytest-dev/pytest-cov/pull/582></code>_.</li> <li>Refactored code to use f-strings. Contributed by Mark Mayo in <code>[#572](pytest-dev/pytest-cov#572) <https://github.com/pytest-dev/pytest-cov/pull/572></code>_.</li> <li>Fixed a skip in the test suite for some old xdist. Contributed by a bunch of people in <code>[#565](pytest-dev/pytest-cov#565) <https://github.com/pytest-dev/pytest-cov/pull/565></code>_.</li> <li>Dropped support for Python 3.6.</li> </ul> <h2>4.0.0 (2022-09-28)</h2> <p><strong>Note that this release drops support for multiprocessing.</strong></p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/95404375a0e436178e012e20b8865c23c54c8a50"><code>9540437</code></a> Bump version: 5.0.0 → 6.0.0</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/9f8175467afc67db9001fb364ad1f2dfe79b51f1"><code>9f81754</code></a> Further trim down envs and drop Python 3.8.</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/b12b5ec65da4c66bbc0c35918957f9f875f465c3"><code>b12b5ec</code></a> Update conf.</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/23f4b27b432a54fcc3b6df7363f0e73e568233fb"><code>23f4b27</code></a> Update changelog.</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/291a04f49566054bb19644aba27c3799ac8b7f42"><code>291a04f</code></a> Bump test deps and trim config.</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/08f1101455ba293dda388fdb3b61e62fd95a827d"><code>08f1101</code></a> Add <code>--cov-precision</code> option. Close <a href="https://redirect.github.com/pytest-dev/pytest-cov/issues/655">#655</a>.</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/76fe2a7e48e5c9c53644994c5ba5a421c84286f5"><code>76fe2a7</code></a> Move the warnings/errors in a place that doesn't import anything.</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/a9ea7b71711479d4c5ccc5e21e2eb1694b259cb0"><code>a9ea7b7</code></a> Implement error/warning for the bad dynamic_context being set in config.</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/c299e01b6422284a6a7f7322e6b9bf8f44aa3c25"><code>c299e01</code></a> Add explicit suffixing to make it easier to see the identify the sources/usag...</li> <li><a href="https://github.com/pytest-dev/pytest-cov/commit/c87e54643ef696ed2b0e2b9a4209581da8467fcb"><code>c87e546</code></a> Add reproducer for weird xdist dynamic_context interaction. Ref <a href="https://redirect.github.com/pytest-dev/pytest-cov/issues/604">#604</a>.</li> <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-cov/compare/v2.10.0...v6.0.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pytest-cov&package-manager=pip&previous-version=2.10.0&new-version=6.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: i_virus <[email protected]>

mpurusottamc and others added 18 commits December 5, 2020 07:47

Updated Readme to add Cloudanix as a Cartography user

9eaba96

Merge remote-tracking branch 'upstream/master' into master

2cd649d

# Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.

Merge branch 'master' of github.com:lyft/cartography

8d33f8d

Merge branch 'master' of github.com:lyft/cartography

5e9d8d2

Merge branch 'master-upstream'

ddc0d06

Merge branch 'master-upstream'

917d702

Merge branch 'master-upstream'

219d3bb

Updates in EC2 codes

3d0ec71

minor documentation updates

d71d11e

Minor changes

7a4a7ab

Added Support EC2 Images (AMIs)

b777924

minor typo fix

4ec04d7

Minor index name change

85dc5cb

Added indentation in UNWIND

35bff10

Merge branch 'master' into aws-ec2-enhancements

68217da

Linter fix

3a46104

Merge branch 'aws-ec2-enhancements' of https://github.com/Cloudanix/c…

b857fcf

…artography into aws-ec2-enhancements

Minor logic and linter fixes

894d120

achantavy reviewed Jun 10, 2021

View reviewed changes

kedarghule added 2 commits June 10, 2021 20:25

Changes as per review

6cf2814

Updated AMI Id attribute as ImageIds are only unique to a region

35d945c

kedarghule requested a review from achantavy June 10, 2021 15:45

Merge branch 'master' into aws-ec2-enhancements

16b713e

achantavy approved these changes Jun 16, 2021

View reviewed changes

achantavy merged commit c5872ef into cartography-cncf:master Jun 16, 2021

ryan-lane mentioned this pull request Jun 17, 2021

Exception in loading autoscale groups #642

Closed

kedarghule mentioned this pull request Apr 29, 2022

"string indices must be integers" error #801

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enhancements in EC2 and Added Support for AMIs #633

Enhancements in EC2 and Added Support for AMIs #633

Uh oh!

kedarghule commented Jun 1, 2021 •

edited

Loading

Uh oh!

achantavy left a comment

Uh oh!

achantavy Jun 10, 2021 •

edited

Loading

Uh oh!

kedarghule Jun 10, 2021

Uh oh!

achantavy Jun 10, 2021

Uh oh!

kedarghule Jun 10, 2021

Uh oh!

achantavy Jun 10, 2021

Uh oh!

achantavy Jun 10, 2021 •

edited

Loading

Uh oh!

kedarghule Jun 10, 2021

Uh oh!

mpurusottamc Jun 10, 2021

Uh oh!

kedarghule Jun 10, 2021

Uh oh!

achantavy Jun 10, 2021

Uh oh!

kedarghule Jun 10, 2021

Uh oh!

kedarghule commented Jun 15, 2021

Uh oh!

achantavy left a comment

Uh oh!

Uh oh!


		## EC2Image

		Representation of an AWS [EC2 Images (AMIs)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html).


		## AutoScalingGroup

		Representation of an AWS [Auto Scaling Group Resource](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html).

		"""

		# neo4j does not accept datetime objects and values. This loop is used to convert

Enhancements in EC2 and Added Support for AMIs #633

Enhancements in EC2 and Added Support for AMIs #633

Uh oh!

Conversation

kedarghule commented Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

achantavy left a comment

Choose a reason for hiding this comment

Uh oh!

achantavy Jun 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

achantavy Jun 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kedarghule commented Jun 15, 2021

Uh oh!

achantavy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kedarghule commented Jun 1, 2021 •

edited

Loading

achantavy Jun 10, 2021 •

edited

Loading

achantavy Jun 10, 2021 •

edited

Loading