Merge pull request #196 from certego/develop

4.1.0

Author: mlodic

.github/CHANGELOG.md

@@ -1,6 +1,34 @@
 # Changelog
 
 [**Upgrade Guide**](https://intelowl.readthedocs.io/en/latest/Installation.md#update-to-the-most-recent-version)
+## [v4.1.5](https://github.com/intelowlproject/IntelOwl/releases/tag/v4.1.5)
+With this release we announce that IntelOwl Project will apply as a new Organization in the next [Google Summer of Code](https://summerofcode.withgoogle.com/)!
+
+We have created a dedicated repository with all the info an aspiring contributor would need to participate to the program.
+
+All open source and cyber security fans! We are calling you! Be the next contributor!
+
+(...and under the hood we did some fixes and updates here and there)
+
+## [v4.1.4](https://github.com/intelowlproject/IntelOwl/releases/tag/v4.1.4)
+With this release we welcome our first sponsor in [Open Collective](https://opencollective.com/intelowl-project): [ThreatHunter.ai](https://threathunter.ai/?utm_source=intelowl)! Thank you for your help!
+
+Moreover this release solves a bug regarding the creation of organization-level secrets which was not possible before.
+
+And this is the last release of this year for us! We will see each other back in 2023!
+
+## [v4.1.3](https://github.com/intelowlproject/IntelOwl/releases/tag/v4.1.3)
+With this version we officially announce that we have joined [Open Collective](https://opencollective.com/intelowl-project) with the IntelOwl Project!
+
+If you love this project and you would like to help us, we would love to get your support there!
+<a href="https://opencollective.com/intelowl-project/donate" target="_blank">
+  <img src="https://opencollective.com/intelowl-project/donate/[email protected]?color=blue" width=150 />
+</a>
+
+**New/Improved Analyzers:**
+* adjusted / fixed a lot of popular analyzers like Dehashed, MISP, VirusTotal, Alienvault OTX, PDF_Info and Unpacme
+* fixed --malware_tools_analyzers broken
+
 ## [v4.1.2](https://github.com/intelowlproject/IntelOwl/releases/tag/v4.1.2)
 
 This version mainly adds quality improvements to the recently released ["Playbook" feature](https://intelowl.readthedocs.io/en/latest/Usage.html#playbooks):

.github/ISSUE_TEMPLATE/new_analyzer.md

@@ -1,6 +1,6 @@
 ---
 name: New Analyzer
-about: A new analyzer to integrate with intel owl
+about: A new analyzer to integrate with IntelOwl
 title: "[Analyzer]"
 labels: new_analyzer
 assignees: ''

.github/ISSUE_TEMPLATE/new_connector.md

@@ -0,0 +1,21 @@
+---
+name: New Connector
+about: A new connector to integrate with IntelOwl
+title: "[Connector]"
+labels: new_connector
+assignees: ''
+
+---
+
+## Name
+
+## Link
+
+## Type of connector
+** what kind of data this connector would push to the integrated service **
+
+
+## Why should we use it
+
+
+## Possible implementation

.github/ISSUE_TEMPLATE/new_ingestor.md

@@ -0,0 +1,18 @@
+---
+name: New Ingestor
+about: A new ingestor to integrate with IntelOwl
+title: "[Ingestor]"
+labels: new_ingestor
+assignees: ''
+
+---
+
+## Name
+
+## Link
+
+
+## Why should we use it
+
+
+## Possible implementation

.github/ISSUE_TEMPLATE/new_scanner.md

@@ -0,0 +1,21 @@
+---
+name: New Scanner
+about: A new scanner to integrate with IntelOwl
+title: "[Scanner]"
+labels: new_scanner
+assignees: ''
+
+---
+
+## Name
+
+## Link
+
+## Type of scanner
+**this can be observable, file, and docker**
+
+
+## Why should we use it
+
+
+## Possible implementation

.github/pull_request_template.md

@@ -2,7 +2,7 @@
 
 # Description
 
-Please include a summary of the change.
+Please include a summary of the change and link to the related issue.
 
 ## Type of change
 
@@ -16,23 +16,21 @@ Please delete options that are not relevant.
 
 - [ ] I have read and understood the rules about [how to Contribute](https://intelowl.readthedocs.io/en/latest/Contribute.html) to this project
 - [ ] The pull request is for the branch `develop`
-- [ ] A new analyzer or connector was added, in which case:
+- [ ] A new plugin (analyzer, connector or playbook) was added or changed, in which case:
     - [ ] [Usage](https://github.com/intelowlproject/IntelOwl/blob/master/docs/source/Usage.md) file was updated.
-    - [ ] [Advanced-Usage](./Advanced-Usage.md) was updated (in case the analyzer/connector provides additional optional configuration).
+    - [ ] [Advanced-Usage](./Advanced-Usage.md) was updated (in case the plugin provides additional optional configuration).
     - [ ] Secrets were added in [env_file_app_template](https://github.com/intelowlproject/IntelOwl/blob/master/docker/env_file_app_template), [env_file_app_ci](https://github.com/certego/IntelOwl/blob/master/docker/env_file_app_ci) and in the [Installation](./Installation.md) docs, if necessary.
-    - [ ] If the analyzer/connector requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
+    - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
     - [ ] If a File analyzer was added, its name was explicitly defined in [test_file_scripts.py](https://github.com/intelowlproject/IntelOwl/blob/master/tests/analyzers_manager/test_file_scripts.py) (not required for Observable Analyzers).
+    - [ ] If you created a new analyzer and it is free (does not require API keys), please add it in the `FREE_TO_USE_ANALYZERS` playbook in `playbook_config.json`
+    - [ ] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
 - [ ] If external libraries/packages with restrictive licenses were used, they were added in the [Legal Notice](https://github.com/certego/IntelOwl/blob/master/.github/legal_notice.md) section.
 - [ ] The tests gave 0 errors.
 - [ ] Linters (`Black`, `Flake`, `Isort`) gave 0 errors. If you have correctly installed [pre-commit](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-start-setup-project-and-development-instance), it does these checks and adjustments on your behalf.
 - [ ] If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check [CONTRIBUTE.md](./Contribute.md)).
-- [ ] If the analyzer is free, Please add it in the `FREE_TO_USE_ANALYZERS` playbook in `playbook_config.json`
+- [ ] If the GUI has been modified, I have a provided a screenshot of the result in the PR.
 
 ### Important Rules
+- If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
 - If your changes decrease the overall tests coverage (you will know after the Codecov CI job is done), you should add the required tests to fix the problem
-- Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review
-
-# Real World Example
-
-Please delete if the PR is for bug fixing.
-Otherwise, please provide the resulting raw JSON of a finished analysis (and, if you like, a screenshot of the results). This is to allow the maintainers to understand how the analyzer works.
+- Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

.github/release_template.md

@@ -2,7 +2,7 @@
 
 - [ ] (optional) If we changed/added Docker Analyzers, we need to configure Docker Hub / Dependabot properly.
 - [ ] Update `CHANGELOG.md` for the new version
-- [ ] Change version number in `docs/source/conf.py`, `docs/schema.yml`, `docker/.env`, `docker/.version`, `intel_owl/settings.commons.py` and `start.py`
+- [ ] Change version number in `docs/schema.yml` and `docker/.env`
 - [ ] Verify CI Tests
 - [ ] Create release for the branch `develop`. A Github action should automatically create a [Twitter](https://twitter.com/intel_owl) post.
       Write the following statement there (change the version number):

.github/workflows/pull_request_automation.yml

@@ -36,7 +36,7 @@ jobs:
         uses: actions/checkout@v3
 
       - name: Set up Python
-        uses: actions/setup-python@v4.3.0
+        uses: actions/setup-python@v4.5.0
         with:
           python-version: 3.9
 
@@ -136,9 +136,9 @@ jobs:
         run: |
           docker/scripts/coverage_test.sh tests.analyzers_manager.test_file_scripts
 
-      - name: "Test: Playbooks Manager (controllers, views)"
+      - name: "Test: Playbooks Manager (views)"
         run: |
-          docker/scripts/coverage_test.sh tests.playbooks_manager.test_controller tests.playbooks_manager.test_views
+          docker/scripts/coverage_test.sh tests.playbooks_manager
 
       - name: "Coverage: generate xml and transfer from docker container to host"
         run: |

.github/workflows/scorecard.yml

@@ -0,0 +1,72 @@
+# This workflow uses actions that are not certified by GitHub. They are provided
+# by a third-party and are governed by separate terms of service, privacy
+# policy, and support documentation.
+
+name: Scorecard supply-chain security
+on:
+  # For Branch-Protection check. Only the default branch is supported. See
+  # https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection
+  branch_protection_rule:
+  # To guarantee Maintained check is occasionally updated. See
+  # https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained
+  schedule:
+    - cron: '23 1 * * 0'
+  push:
+    branches: [ "master" ]
+
+# Declare default permissions as read only.
+permissions: read-all
+
+jobs:
+  analysis:
+    name: Scorecard analysis
+    runs-on: ubuntu-latest
+    permissions:
+      # Needed to upload the results to code-scanning dashboard.
+      security-events: write
+      # Needed to publish results and get a badge (see publish_results below).
+      id-token: write
+      # Uncomment the permissions below if installing in a private repository.
+      # contents: read
+      # actions: read
+
+    steps:
+      - name: "Checkout code"
+        uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
+        with:
+          persist-credentials: false
+
+      - name: "Run analysis"
+        uses: ossf/scorecard-action@e38b1902ae4f44df626f11ba0734b14fb91f8f86 # v2.1.2
+        with:
+          results_file: results.sarif
+          results_format: sarif
+          # (Optional) "write" PAT token. Uncomment the `repo_token` line below if:
+          # - you want to enable the Branch-Protection check on a *public* repository, or
+          # - you are installing Scorecard on a *private* repository
+          # To create the PAT, follow the steps in https://github.com/ossf/scorecard-action#authentication-with-pat.
+          # repo_token: ${{ secrets.SCORECARD_TOKEN }}
+
+          # Public repositories:
+          #   - Publish results to OpenSSF REST API for easy access by consumers
+          #   - Allows the repository to include the Scorecard badge.
+          #   - See https://github.com/ossf/scorecard-action#publishing-results.
+          # For private repositories:
+          #   - `publish_results` will always be set to `false`, regardless
+          #     of the value entered here.
+          publish_results: true
+
+      # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
+      # format to the repository Actions tab.
+      - name: "Upload artifact"
+        uses: actions/upload-artifact@3cea5372237819ed00197afe530f5a7ea3e805c8 # v3.1.0
+        with:
+          name: SARIF file
+          path: results.sarif
+          retention-days: 5
+
+      # Upload the results to GitHub's code scanning dashboard.
+      - name: "Upload to code-scanning"
+        uses: github/codeql-action/upload-sarif@807578363a7869ca324a79039e6db9c843e0e100 # v2.1.27
+        with:
+          sarif_file: results.sarif

.github/workflows/sphinx_build_on_release.yml

@@ -13,7 +13,7 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python
-        uses: actions/setup-python@v4.3.0
+        uses: actions/setup-python@v4.5.0
         with:
           python-version: 3.9
       - name: Install docs requirements