fixes #1699 Permhash analyzer

[moonpatel]

(Please add to the PR name the issue/s that this PR would close if merged by using a Github keyword. Example: <feature name>. Closes #999. If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)

Description

Closes #1699
New analyzer: Permhash

Type of change

New analyzer

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue).
• New feature (non-breaking change which adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

• I have read and understood the rules about how to Contribute to this project
• The pull request is for the branch develop
• A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
  • I strictly followed the documentation "How to create a Plugin"
  • Usage file was updated.
  • Advanced-Usage was updated (in case the plugin provides additional optional configuration).
  • If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
  • I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. ("How to share a plugin with the community")
  • If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
  • If you created a new analyzer and it is free (does not require API keys), please add it in the FREE_TO_USE_ANALYZERS playbook by following this guide.
  • Check if it could make sense to add that analyzer/connector to other freely available playbooks.
  • I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
  • I have added that raw JSON sample to the MockUpResponse of the _monkeypatch() method. This serves us to provide a valid sample for testing.
• If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
• If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
• Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
• I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
• If the GUI has been modified:
  • I have a provided a screenshot of the result in the PR.
  • I have created new frontend tests for the new component or updated existing ones.
• After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

Important Rules

• If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
• Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

Output screenshot

JSON output

{
    "hash":"aad106ceb64ac2a636ddec77c3feed4c2ffc5c27ab353660d8cb3e1c971df277"
}

[moonpatel]

Hey @mlodic I have added new mimetypes for .crx and .json but migration shows error.
Here are the migration error logs:

intelowl_uwsgi                  |   Applying analyzers_manager.0066_analyzer_config_phoneinfoga... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0067_update_misp... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0068_analyzer_config_feodo_tracker... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0069_analyzer_config_bgp_ranking... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0070_urlhaus_threatfox_disable_param... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0071_analyzer_config_tor_nodes_danmeuk... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0072_analyzer_config_tweetfeed... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0073_remove_dragonfly_analyzer... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0074_adjust_maximum_tlp... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0075_adjust_greynoise... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0076_analyzer_config_greynoise_labs... OK
intelowl_uwsgi                  |   Applying analyzers_manager.0077_analyzer_config_abusix... OK
intelowl_uwsgi                  | Traceback (most recent call last):
intelowl_uwsgi                  |   File "/opt/deploy/intel_owl/api_app/analyzers_manager/migrations/0078_analyzer_config_permhash.py", line 54, in _create_object
intelowl_uwsgi                  |     o = Model.objects.get(**no_mtm)
intelowl_uwsgi                  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
intelowl_uwsgi                  |     return getattr(self.get_queryset(), name)(*args, **kwargs)
intelowl_uwsgi                  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
intelowl_uwsgi                  |     raise self.model.DoesNotExist(
intelowl_uwsgi                  | __fake__.AnalyzerConfig.DoesNotExist: AnalyzerConfig matching query does not exist.
intelowl_uwsgi                  | 
intelowl_uwsgi                  | During handling of the above exception, another exception occurred:
intelowl_uwsgi                  | 
intelowl_uwsgi                  | Traceback (most recent call last):
intelowl_uwsgi                  |   File "/opt/deploy/intel_owl/manage.py", line 25, in <module>
intelowl_uwsgi                  |     main()
intelowl_uwsgi                  |   File "/opt/deploy/intel_owl/manage.py", line 21, in main
intelowl_uwsgi                  |     execute_from_command_line(sys.argv)
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
intelowl_uwsgi                  |     utility.execute()
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
intelowl_uwsgi                  |     self.fetch_command(subcommand).run_from_argv(self.argv)
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
intelowl_uwsgi                  |     self.execute(*args, **cmd_options)
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
intelowl_uwsgi                  |     output = self.handle(*args, **options)
intelowl_uwsgi                  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
intelowl_uwsgi                  |     res = handle_func(*args, **kwargs)
intelowl_uwsgi                  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/opt/deploy/intel_owl/api_app/management/commands/migrate.py", line 8, in handle
intelowl_uwsgi                  |     super().handle(*args, **options)
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
intelowl_uwsgi                  |     res = handle_func(*args, **kwargs)
intelowl_uwsgi                  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
intelowl_uwsgi                  |     post_migrate_state = executor.migrate(
intelowl_uwsgi                  |                          ^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
intelowl_uwsgi                  |     state = self._migrate_all_forwards(
intelowl_uwsgi                  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
intelowl_uwsgi                  |     state = self.apply_migration(
intelowl_uwsgi                  |             ^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
intelowl_uwsgi                  |     state = migration.apply(state, schema_editor)
intelowl_uwsgi                  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
intelowl_uwsgi                  |     operation.database_forwards(
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
intelowl_uwsgi                  |     self.code(from_state.apps, schema_editor)
intelowl_uwsgi                  |   File "/opt/deploy/intel_owl/api_app/analyzers_manager/migrations/0078_analyzer_config_permhash.py", line 72, in migrate
intelowl_uwsgi                  |     exists = _create_object(Model, plugin)
intelowl_uwsgi                  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi                  |   File "/opt/deploy/intel_owl/api_app/analyzers_manager/migrations/0078_analyzer_config_permhash.py", line 57, in _create_object
intelowl_uwsgi                  |     o.full_clean()
intelowl_uwsgi                  |   File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1502, in full_clean
intelowl_uwsgi                  |     raise ValidationError(errors)
intelowl_uwsgi                  | django.core.exceptions.ValidationError: {'supported_filetypes': ["Item 6 in the array did not validate: Value 'application/x-chrome-extension' is not a valid choice."]}
intelowl_uwsgi                  |   Applying analyzers_manager.0078_analyzer_config_permhash...Issue with migration exiting
intelowl_uwsgi exited with code 1
dependency failed to start: container intelowl_uwsgi exited (1)

[mlodic]

you are changing a model, you need to do a migration for this change too

[mlodic]

why check the file extension when you can use the mimetype?

[mlodic]

you should raise the AnalyzerRunException

[mlodic]

you should raise the AnalyzerRunException

[moonpatel]

======================================================================
FAIL: test_subclasses (tests.api_app.analyzers_manager.test_classes.FileAnalyzerTestCase.test_subclasses)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/deploy/intel_owl/tests/api_app/analyzers_manager/test_classes.py", line 133, in test_subclasses
    sub.start(job.pk, {}, uuid())
  File "/usr/local/lib/python3.11/unittest/mock.py", line 1375, in patched
    return func(*newargs, **newkeywargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/deploy/intel_owl/api_app/classes.py", line 208, in start
    self.after_run_failed(e)
  File "/opt/deploy/intel_owl/api_app/classes.py", line 153, in after_run_failed
    raise e
  File "/opt/deploy/intel_owl/api_app/classes.py", line 206, in start
    _result = self.run()
              ^^^^^^^^^^
  File "/opt/deploy/intel_owl/api_app/analyzers_manager/file_analyzers/perm_hash.py", line 52, in run
    raise AnalyzerRunException("Could not find permissions in the file.")
api_app.analyzers_manager.exceptions.AnalyzerRunException: Could not find permissions in the file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/deploy/intel_owl/tests/api_app/analyzers_manager/test_classes.py", line 135, in test_subclasses
    self.fail(
AssertionError: Analyzer Permhash with config Permhash and mimetype application/vnd.android.package-archive failed Could not find permissions in the file.

----------------------------------------------------------------------
Ran 312 tests in 210.989s

FAILED (failures=1, skipped=1)
Preserving test database for alias 'default'...
Error: Process completed with exit code 1.

I think we should not raise the exception here as it fails the test cases
OR
we can replace the sample.apk in the test_files.zip with an apk that has an AndroidManifest.xml with permissions included as the current sample.apk's manifest file has no such thing. But replacing that apk may led to some test failures in other file analyzers. What are your thoughts @mlodic ?

[moonpatel]

This analyzer is not working when I tried testing it in the frontend. This is what I get (although other analyzers are working fine):

[mlodic]

if you don't show the logs I don't know how to help. Also, please pull from develop that you have conflicts

[mlodic]

please write also a reverse_migrate function. That helps to recover in case of mistakes

[mlodic]

please also add this analyzer to the freely available playbook + to the static analyzer playbook

[mlodic]

you also need to change the test_classes.py if you add a new filetype and add a new test file for that type

[moonpatel]

2024-04-11 17:20:39,528 - api_app.models - set_final_status - INFO - <Job: Job(#1, "12.1.2.3")> setting status to reported_without_fails
2024-04-11 17:20:39,534 - api_app.websocket - serialize_and_send_job - DEBUG - send data for the job: 1 to the groups: ['job-1', 'job-1-perm']
2024-04-11 17:20:39,535 - api_app.websocket - serialize_and_send_job - DEBUG - send data to the group: job-1
2024-04-11 17:20:39,622 - api_app.websocket - serialize_and_send_job - DEBUG - send data to the group: job-1-perm
2024-04-11 17:21:54,451 - api_app.views - analyze_multiple_files - INFO - received analyze_multiple_files from user moon
2024-04-11 17:21:54,465 - api_app.serializers.job - validate - DEBUG - before attrs: OrderedDict([('user', <SimpleLazyObject: <User: moon>>), ('is_sample', True), ('tlp', 'AMBER'), ('analyzers_requested', [<AnalyzerConfig: Permhash>]), ('connectors_requested', []), ('tags_labels', <generator object _AbstractJobCreateSerializer.validate_tags_labels at 0x7fe69c27f840>), ('scan_mode', 2), ('scan_check_time', datetime.timedelta(days=1)), ('investigation', None), ('file', <InMemoryUploadedFile: bitbar-sample-app.apk (application/vnd.android.package-archive)>)])
2024-04-11 17:21:54,532 - api_app.serializers.job - validate - DEBUG - after attrs: OrderedDict([('user', <SimpleLazyObject: <User: moon>>), ('is_sample', True), ('tlp', 'AMBER'), ('analyzers_requested', [<AnalyzerConfig: Permhash>]), ('connectors_requested', []), ('scan_mode', 2), ('scan_check_time', datetime.timedelta(days=1)), ('investigation', None), ('file', <InMemoryUploadedFile: bitbar-sample-app.apk (application/vnd.android.package-archive)>), ('file_name', 'bitbar-sample-app.apk'), ('file_mimetype', MimeTypes.APK), ('md5', '00cc5435151aa38a091781922c0390a4'), ('analyzers_to_execute', [<AnalyzerConfig: Permhash>]), ('connectors_to_execute', []), ('visualizers_to_execute', []), ('warnings', []), ('tags', <generator object _AbstractJobCreateSerializer.validate_tags_labels at 0x7fe69c27f840>)])
2024-04-11 17:21:54,533 - api_app.serializers.job - check_previous_jobs - INFO - Checking previous jobs
2024-04-11 17:21:54,604 - api_app.serializers.job - create - INFO - Job 2 created
2024-04-11 17:21:54,604 - api_app.serializers.job - create - INFO - Sending task for job 2
2024-04-11 17:21:54,626 - api_app.views - analyze_multiple_files - INFO - finished analyze_multiple_files from user moon
2024-04-11 17:21:55,943 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>
2024-04-11 17:21:56,387 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>
2024-04-11 17:22:17,056 - api_app.mixins - list - DEBUG - View list_ConnectorConfig_moon_1_70 cache hit
2024-04-11 17:22:17,085 - api_app.mixins - list - DEBUG - View list_AnalyzerConfig_moon_1_70 cache hit
2024-04-11 17:22:17,185 - api_app.mixins - list - DEBUG - View list_PivotConfig_moon_1_70 cache hit
2024-04-11 17:22:17,317 - api_app.mixins - list - DEBUG - View list_VisualizerConfig_moon_1_70 cache hit
2024-04-11 17:22:17,419 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>
2024-04-11 17:22:17,478 - api_app.mixins - list - DEBUG - View list_IngestorConfig_moon_1_70 cache hit
2024-04-11 17:22:17,584 - api_app.mixins - list - DEBUG - View list_AnalyzerConfig_moon_2_70 cache hit
2024-04-11 17:22:17,760 - api_app.mixins - list - DEBUG - View list_AnalyzerConfig_moon_3_70 cache hit
2024-04-11 17:22:17,864 - api_app.mixins - list - DEBUG - View list_PlaybookConfig_moon_1_70 cache hit
2024-04-11 17:22:21,377 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>

@mlodic

[moonpatel]

Actually, I solved it myself. There was some issue with the latest version of permhash library (v0.1.4.2). I used v0.1.4 and it worked perfectly fine.