Skip to content

Folder monitor #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 29 commits into
base: mainline
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
28e8f57
Initial work on FileSystemIO
ajCameron Jul 4, 2022
0db0e88
Adding more comprehensive tests to fix the post run file creation issues
ajCameron Jul 5, 2022
3cfdedb
About to completely re-write using asyncinotify instead of watchfiles
ajCameron Jul 6, 2022
f5dad36
Prototype is in a functional state
ajCameron Jul 11, 2022
ab7309d
Broken file_input down into compoents
ajCameron Jul 11, 2022
246e502
Added explicit events for target object creation
ajCameron Jul 12, 2022
bb234df
Encountered an annoying bug during testing
ajCameron Jul 12, 2022
c1d5113
windows input is now tested and working
ajCameron Jul 21, 2022
c464921
Output tested and working
ajCameron Jul 22, 2022
d4e66d3
Added and tested a linux file monitor
ajCameron Jul 25, 2022
71626a3
Updating tests and monitors to try and produce (somewhat similar resu…
ajCameron Jul 25, 2022
5bbe610
Additional tests for the linux file monitor
ajCameron Jul 26, 2022
5d31c81
Windows side tests are passing
ajCameron Jul 27, 2022
517695c
File input mode is now being properly tested on linux
ajCameron Jul 27, 2022
1c7ff21
File input mode is now being properly tested on linux
ajCameron Jul 28, 2022
9b65d04
Testing complete
ajCameron Jul 30, 2022
ae6a222
Modified test due to _yet another_ unexpected case
ajCameron Jul 30, 2022
2623cf2
Commit that will probably be re-write
ajCameron Jul 30, 2022
a74ed1a
here we go again
ajCameron Jul 31, 2022
172190e
Re-writing to DRY the code base
ajCameron Aug 1, 2022
10968ba
Refactoring code to enable deletion of specific test cases
ajCameron Aug 1, 2022
0fa358c
Timeout added to queue get
ajCameron Aug 3, 2022
f1c9784
Linux versions of the tests now working after the refactor.
ajCameron Aug 3, 2022
89d25c8
DRYing out monitors.py
ajCameron Aug 4, 2022
74a48b2
Continuing working towards Sonarcloud compliance
ajCameron Aug 4, 2022
de13df7
Attempted fix to address Sonarcloud bugs
ajCameron Aug 4, 2022
aa18b10
Sonarcloud line duplicate requirements may be met
ajCameron Aug 4, 2022
0352ee0
Merge branch 'mainline' into folder_monitor
ajCameron Apr 9, 2023
11046c9
Merge branch 'mainline' into folder_monitor
ajCameron Jul 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,4 @@ examples/TrivialDiscordBot/secrets.json
/reports/
/.pytest_cache/
/junit.xml
.coverage*
1 change: 1 addition & 0 deletions dev-docs/developer-notes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Catch all place for individual developers to store notes they don't mind being public
11 changes: 11 additions & 0 deletions dev-docs/developer-notes/cameron-general-notes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@

https://github.com/topics/fediverse?l=python

Standardization and upgrade run - next step

- standardize all setter values to value
- There's a few places - e.g. in FileSystem - where strings could be enums for clarity
- And type(self).__name__ could be used in a bunch of different places to save on changes after a class rename
- Consistent and present docstrings everywhere


6 changes: 6 additions & 0 deletions dev-docs/io-dev-notes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
The aim of this folder is to include notes written by the developers as they were working.
These will - probably - eventually - end up on some kind of wiki.

Feel free to write as much (or as little) as you like.

Personally, these are my working notes. So expect some digressions and random other useful stuff.
354 changes: 354 additions & 0 deletions dev-docs/io-dev-notes/file_system-dev-notes.md

Large diffs are not rendered by default.

Empty file.
110 changes: 110 additions & 0 deletions examples/file_system_bots/dir_input_monitor_bot.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
#!/usr/bin/env python3
# pylint: disable=duplicate-code
# this is an example - duplication for emphasis is desirable

from __future__ import annotations

from typing import Any, Dict, Set, Type

import logging

from mewbot.api.v1 import Trigger, Action
from mewbot.core import InputEvent, OutputEvent, OutputQueue
from mewbot.io.file_system import (
CreatedDirFSInputEvent,
UpdatedDirFSInputEvent,
MovedDirFSInputEvent,
DeletedDirFSInputEvent,
CreatedFileFSInputEvent,
UpdatedFileFSInputEvent,
MovedFileFSInputEvent,
DeletedFileFSInputEvent,
InputFileDirCreationInputEvent,
InputFileDirDeletionInputEvent,
)


class DirSystemAllCommandTrigger(Trigger):
"""
Nothing fancy - just fires whenever there is a dir related FSInputEvent
"""

@staticmethod
def consumes_inputs() -> Set[Type[InputEvent]]:
return {
CreatedDirFSInputEvent,
UpdatedDirFSInputEvent,
MovedDirFSInputEvent,
DeletedDirFSInputEvent,
CreatedFileFSInputEvent,
UpdatedFileFSInputEvent,
MovedFileFSInputEvent,
DeletedFileFSInputEvent,
InputFileDirCreationInputEvent,
InputFileDirDeletionInputEvent,
}

def matches(self, event: InputEvent) -> bool:

print("-------\n", "event seen by matches - ", event, "\n-------")

if not isinstance(
event,
(
CreatedDirFSInputEvent,
UpdatedDirFSInputEvent,
MovedDirFSInputEvent,
DeletedDirFSInputEvent,
CreatedFileFSInputEvent,
MovedFileFSInputEvent,
UpdatedFileFSInputEvent,
DeletedFileFSInputEvent,
InputFileDirCreationInputEvent,
InputFileDirDeletionInputEvent,
),
):
return False

return True


class DirSystemInputPrintResponse(Action):
"""
Print every DirSystem Dir related InputEvent.
"""

_logger: logging.Logger
_queue: OutputQueue

def __init__(self) -> None:
super().__init__()
self._logger = logging.getLogger(__name__ + type(self).__name__)

@staticmethod
def consumes_inputs() -> Set[Type[InputEvent]]:
return {
CreatedDirFSInputEvent,
UpdatedDirFSInputEvent,
MovedDirFSInputEvent,
DeletedDirFSInputEvent,
CreatedFileFSInputEvent,
UpdatedFileFSInputEvent,
MovedFileFSInputEvent,
DeletedFileFSInputEvent,
InputFileDirCreationInputEvent,
InputFileDirDeletionInputEvent,
}

@staticmethod
def produces_outputs() -> Set[Type[OutputEvent]]:
return set()

async def act(self, event: InputEvent, state: Dict[str, Any]) -> None:
"""
Construct a DiscordOutputEvent with the result of performing the calculation.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Construct a DiscordOutputEvent with the result of performing the calculation.
Print a logging message

"""
if not isinstance(event, InputEvent):
self._logger.warning("Received wrong event type %s", type(event))
return

print("-------\n", "event seen by action - ", event, "\n-------")
24 changes: 24 additions & 0 deletions examples/file_system_bots/dir_input_monitor_bot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
kind: IOConfig
implementation: mewbot.io.file_system.FileSystemIO
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa00
properties:
input_path: C:\mewbot_dev\new_dir
input_path_type: dir
---

kind: Behaviour
implementation: mewbot.api.v1.Behaviour
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa01
properties:
name: 'Target Dir Change Alerts'
triggers:
- kind: Trigger
implementation: examples.file_system_bots.dir_input_monitor_bot.DirSystemAllCommandTrigger
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa02
properties: {}
conditions: []
actions:
- kind: Action
implementation: examples.file_system_bots.dir_input_monitor_bot.DirSystemInputPrintResponse
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa03
properties: {}
23 changes: 23 additions & 0 deletions examples/file_system_bots/file_input_None_path_monitor_bot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
kind: IOConfig
implementation: mewbot.io.file_system.FileSystemIO
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa00
properties:
input_path: None
---

kind: Behaviour
implementation: mewbot.api.v1.Behaviour
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa01
properties:
name: 'Target File Change Alerts - but with None path - so it just sits here - hopefully not eating all resources'
triggers:
- kind: Trigger
implementation: examples.file_system_bots.file_input_monitor_bot.FileSystemAllCommandTrigger
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa02
properties: {}
conditions: []
actions:
- kind: Action
implementation: examples.file_system_bots.file_input_monitor_bot.FileSystemInputPrintResponse
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa03
properties: {}
81 changes: 81 additions & 0 deletions examples/file_system_bots/file_input_monitor_bot.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
# pylint: disable=duplicate-code
# this is an example - duplication for emphasis is desirable

from __future__ import annotations

from typing import Any, Dict, Set, Type

import logging

from mewbot.api.v1 import Trigger, Action
from mewbot.core import InputEvent, OutputEvent, OutputQueue
from mewbot.io.file_system import (
FSInputEvent,
UpdatedFileFSInputEvent,
InputFileFileCreationInputEvent,
InputFileFileDeletionInputEvent,
)


class FileSystemAllCommandTrigger(Trigger):
"""
Nothing fancy - just fires whenever there is a file related FSInputEvent
"""

@staticmethod
def consumes_inputs() -> Set[Type[InputEvent]]:
return {
InputFileFileCreationInputEvent,
UpdatedFileFSInputEvent,
InputFileFileDeletionInputEvent,
}

def matches(self, event: InputEvent) -> bool:

if not isinstance(
event,
(
InputFileFileCreationInputEvent,
UpdatedFileFSInputEvent,
InputFileFileDeletionInputEvent,
),
):
return False

return True


class FileSystemInputPrintResponse(Action):
"""
Print every FileSystem File related InputEvent.
"""

_logger: logging.Logger
_queue: OutputQueue

def __init__(self) -> None:
super().__init__()
self._logger = logging.getLogger(__name__ + type(self).__name__)

@staticmethod
def consumes_inputs() -> Set[Type[InputEvent]]:
return {
InputFileFileCreationInputEvent,
UpdatedFileFSInputEvent,
InputFileFileDeletionInputEvent,
}

@staticmethod
def produces_outputs() -> Set[Type[OutputEvent]]:
return set()

async def act(self, event: InputEvent, state: Dict[str, Any]) -> None:
"""
Construct a DiscordOutputEvent with the result of performing the calculation.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Construct a DiscordOutputEvent with the result of performing the calculation.
Print a logging message of the event is appropriate.

"""
if not isinstance(event, FSInputEvent):
self._logger.warning("Received wrong event type %s", type(event))
return

print(event)
24 changes: 24 additions & 0 deletions examples/file_system_bots/file_input_monitor_bot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
kind: IOConfig
implementation: mewbot.io.file_system.FileSystemIO
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa00
properties:
input_path: 'C:\mewbot_dev\test_file.md'
input_path_type: 'file'
---

kind: Behaviour
implementation: mewbot.api.v1.Behaviour
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa01
properties:
name: 'Target File Change Alerts'
triggers:
- kind: Trigger
implementation: examples.file_system_bots.file_input_monitor_bot.FileSystemAllCommandTrigger
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa02
properties: {}
conditions: []
actions:
- kind: Action
implementation: examples.file_system_bots.file_input_monitor_bot.FileSystemInputPrintResponse
uuid: aaaaaaaa-aaaa-4aaa-0001-aaaaaaaaaa03
properties: {}
Loading