How to move forward with anitopy?

[FichteFoll]

Anitopy integration is still in an experimental stage. There are some known issues with the parsing that specifically the mpris tracker doesn't handle well (since it doesn't catch exceptions), so I'm not confident just yet to make it the default. It also requires a Python dependency that is not available on many Linux distros (I only checked debian-based). However, even if you want to use it currently in its experimental state, you need to edit the config file manually and change a mostly undocumented setting, which is not good UX.

The following things are needed to make anitopy the default parser:

• Fix some known issues in anitopy, perhaps by forking it since it seems mostly unmaintained (via Feat: add type-hinting for the public API igorcmoura/anitopy#16 (comment)) (I have a list of excepion tracebacks for things anitopy can't parse currently and there are open issues on the repo)
• Add a fallback mechanism to the AnimeInfoExtractor implementation if anitopy cannot be imported
• Add error handling to mpris tracker

The following things are needed to make anitopy a viable alternative if it should not become the default:

• Make the parser configurable via Qt UI (3ed4374)
• Make the parser configurable via Gtk UI

[z411]

I'm planning to release a new point release today so people don't have to pull from master to get the latest changes. For that I'd like to make the parser configurable as a first step; I'll work to add the option in the GTK/Qt UIs for now.

[erengy]

I have a list of excepion tracebacks for things anitopy can't parse currently

May I have a list of these so that I can possibly add them as test cases to Anitomy?

As a side note, would it be viable for Trackma to use Anitomy instead? Not sure about the implications of depending on an external binary, but I've been rewriting Anitomy for some time and the next version is going to have a CLI that can output parsed data as JSON, which I assume could be helpful for projects that cannot directly use the C++ library.

[FichteFoll]

1. strips some special characters e.g. + in Cardfight!! Vanguard - will+Dress
2. Parser warning: Unable to parse episode number '06v1.5' of: Mahoutsukai Reimeiki E06v1.5 [1080p][AAC][JapDub][GerEngSub][Web-DL].mkv
3. Parser warning: Unable to parse episode number '1a' of: [Some-Stuffs]_Jojo's_Bizarre_Adventure_Diamond_is_Unbreakable_NCOP1a_(1920x1080_Blu-Ray_FLAC)_[343E8357].mkv

[D] Tracker (MPRIS): Guessing filename: [GLORY] NieR Automata Ver1.1a - 01 [1080p HEVC]
…
File "/home/fichte/code/trackma/trackma/parser/anitopy.py", line 63, in getEpisode
  return int(self.episode_number)
ValueError: invalid literal for int() with base 10: '1a'

[D] Tracker (MPRIS): Guessing filename: Tokyo Mew Mew New S01E12-[1080p][BDRIP][x265.FLAC].mkv
Tracker (MPRIS) warning: Not playing the next episode of TOKYO MEW MEW NEW (expected: 12, found: 1). Ignoring.

[D] Tracker (MPRIS): Guessing filename: 2.5 Jigen no Ririsa E01 [1080p][E-AC-3][JapDub][GerSub][Web-DL].mkv
…
File "/home/fichte/code/trackma/trackma/tracker/tracker.py", line 263, in _get_playing_show
  (show_title, show_ep) = (aie.getName(), aie.getEpisode())
                                          ^^^^^^^^^^^^^^^^
File "/home/fichte/code/trackma/trackma/parser/anitopy.py", line 63, in getEpisode
  return int(self.episode_number)
         ^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '2.5'

[D] Tracker (MPRIS): Guessing filename: watch?v=KXC6CZyO_6A&pp=ygUQZmlsbGlhbiBha2luYXRvcg==
…
ValueError: invalid literal for int() with base 10: '6A'

[D] Tracker (MPRIS): Guessing filename: NEET Kunoichi to Naze ka Dousei Hajimemashita E01-E02 [1080p][AAC][JapDub][GerSub][Web-DL].mkv
…
ValueError: invalid literal for int() with base 10: '01-E02'

[D] Tracker (MPRIS): Guessing filename: e8cf4075112ea1a0.mp4
ValueError: invalid literal for int() with base 10: '8cf4075112ea1a0'

[D] Tracker (MPRIS): Guessing filename: Ore wa Seikan Kokka no Akutoku Ryoushu! E1P [1080p][AAC][JapDub][GerEngSub][Web-DL].mkv
File "/home/fichte/code/trackma/trackma/tracker/tracker.py", line 263, in _get_playing_show
  (show_title, show_ep) = (aie.getName(), aie.getEpisode())
                                          ~~~~~~~~~~~~~~^^
File "/home/fichte/code/trackma/trackma/parser/anitopy.py", line 64, in getEpisode
  return int(self.episode_number)
ValueError: invalid literal for int() with base 10: '1P'

(this is obviously an incorrect file name, but it should not result in an exception)

internal to trackma:

[D] Tracker (MPRIS): Guessing filename: Arknights 2024 Special Commemorative Animation PV
…
Exception in thread Thread-2 (observe):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 245, in observe
    self.msg.info("Using MPRIS.")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 240, in observe_async
    task.cancel()
^^^^^^^^^^^^^^^^^^
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 235, in observe_async
    try:
        ^
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 105, in properties_watcher
    handle_properties_changed(tracker, msg)
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 115, in handle_properties_changed
    tracker.on_playback_status_change(sender, playback_status)
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 283, in on_playback_status_change
    player.playback_status = playback_status
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 305, in _handle_player_update
    elif player.playback_status == PlaybackStatus.PLAYING:
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fichte/code/trackma/trackma/tracker/mpris.py", line 314, in _update_active_player
    if player.filename != self.last_filename:
                              ^^^^^^^^^^^^^^^^
  File "/home/fichte/code/trackma/trackma/tracker/tracker.py", line 262, in _get_playing_show
    aie = self.parser_class(self.msg, filename)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fichte/code/trackma/trackma/parser/anitopy.py", line 49, in __init__
    self.anime_title = self.__extractAnimeTitle(data)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fichte/code/trackma/trackma/parser/anitopy.py", line 185, in __extractAnimeTitle
    for t in anitype:
             ^^^^^^^
UnboundLocalError: cannot access local variable 'anitype' where it is not associated with a value

[FichteFoll]

As a side note, would it be viable for Trackma to use Anitomy instead? Not sure about the implications of depending on an external binary, but I've been rewriting Anitomy for some time and the next version is going to have a CLI that can output parsed data as JSON, which I assume could be helpful for projects that cannot directly use the C++ library.

Python can use FFI, but using a JSON-based CLI would definitely be easier and also more portable. In general, we will probably keep bundling our primitive parser as a fallback in case the external Python lib anitopy or the external binary for anitomy are not available, but in general having a shared project in the form of anitomy will be most beneficial to reuse existing logic in our projects, imho.

However, getting the external binary to the user's system will be the most problematic part since trackma is cross-platform and there are many Linux distros out there. My experience with C and C++ projects is quite limited, but as long as there are very few dependencies that anitomy has of itself, bundling an executable might be an option. Or just distributing it separately, e.g. via the AUR. It's not something I am comfortable looking into in the near future with my current availability of free time, however.

[erengy]

Thanks for the list! Looks like most of the issues are already resolved in v2.

1. No issues here, as v2 has a better title building algorithm.

Output

>anitomy "Cardfight!! Vanguard - will+Dress"
┌─────────────────────────────────────────────┐
│ Element │ Value                             │
│─────────────────────────────────────────────│
│ title   │ Cardfight!! Vanguard - will+Dress │
└─────────────────────────────────────────────┘

2. Nyaa shows just 6 results for v1.5 and none of them are in the same form. Probably doesn't justify adding a special case. 5 being parsed as episode title looks weird, though.

Output

>anitomy "Mahoutsukai Reimeiki E06v1.5 [1080p][AAC][JapDub][GerEngSub][Web-DL].mkv"
┌─────────────────────────────────────────┐
│ Element          │ Value                │
│─────────────────────────────────────────│
│ title            │ Mahoutsukai Reimeiki │
│ episode          │ 06                   │
│ release_version  │ 1                    │
│ episode_title    │ 5                    │
│ video_resolution │ 1080p                │
│ audio_term       │ AAC                  │
│ release_group    │ JapDub               │
│ source           │ Web-DL               │
│ file_extension   │ mkv                  │
└─────────────────────────────────────────┘

3. Anitomy doesn't parse NCOP1a at the moment.

Output

>anitomy "[Some-Stuffs]_Jojo's_Bizarre_Adventure_Diamond_is_Unbreakable_NCOP1a_(1920x1080_Blu-Ray_FLAC)_[343E8357].mkv"
┌───────────────────────────────────────────────────────────────────────────┐
│ Element          │ Value                                                  │
│───────────────────────────────────────────────────────────────────────────│
│ release_group    │ Some-Stuffs                                            │
│ title            │ Jojo's Bizarre Adventure Diamond is Unbreakable NCOP1a │
│ video_resolution │ 1920x1080                                              │
│ source           │ Blu-Ray                                                │
│ audio_term       │ FLAC                                                   │
│ file_checksum    │ 343E8357                                               │
│ file_extension   │ mkv                                                    │
└───────────────────────────────────────────────────────────────────────────┘

4. This is fixed in v2. See also erengy/taiga#1110 and erengy/anitomy#20.

Output

>anitomy "[GLORY] NieR Automata Ver1.1a - 01 [1080p HEVC]"
┌──────────────────────────────────────────┐
│ Element          │ Value                 │
│──────────────────────────────────────────│
│ release_group    │ GLORY                 │
│ title            │ NieR Automata Ver1.1a │
│ episode          │ 01                    │
│ video_resolution │ 1080p                 │
│ video_term       │ HEVC                  │
└──────────────────────────────────────────┘

5. v2 no longer trips on spaceless dashes. See also erengy/taiga#1222 and other linked issues.

Output

>anitomy "Tokyo Mew Mew New S01E12-[1080p][BDRIP][x265.FLAC].mkv"
┌──────────────────────────────────────┐
│ Element          │ Value             │
│──────────────────────────────────────│
│ title            │ Tokyo Mew Mew New │
│ season           │ 01                │
│ episode          │ 12                │
│ video_resolution │ 1080p             │
│ source           │ BDRIP             │
│ video_term       │ x265              │
│ audio_term       │ FLAC              │
│ file_extension   │ mkv               │
└──────────────────────────────────────┘

6. This is fixed in v2 too.

Output

>anitomy "2.5 Jigen no Ririsa E01 [1080p][E-AC-3][JapDub][GerSub][Web-DL].mkv"
┌────────────────────────────────────────┐
│ Element          │ Value               │
│────────────────────────────────────────│
│ title            │ 2.5 Jigen no Ririsa │
│ episode          │ 01                  │
│ video_resolution │ 1080p               │
│ audio_term       │ E-AC-3              │
│ release_group    │ JapDub              │
│ source           │ Web-DL              │
│ file_extension   │ mkv                 │
└────────────────────────────────────────┘

7. Not a valid filename, so ignoring this one.

Output

>anitomy "watch?v=KXC6CZyO_6A&pp=ygUQZmlsbGlhbiBha2luYXRvcg=="
┌─────────────────────────────────────────────────┐
│ Element       │ Value                           │
│─────────────────────────────────────────────────│
│ title         │ watch?v=KXC6CZyO                │
│ episode       │ 6A                              │
│ episode_title │ pp=ygUQZmlsbGlhbiBha2luYXRvcg== │
└─────────────────────────────────────────────────┘

8. v2 works better but needs some work on parsing multiple episodes.

Output

>anitomy "NEET Kunoichi to Naze ka Dousei Hajimemashita E01-E02 [1080p][AAC][JapDub][GerSub][Web-DL].mkv"
┌──────────────────────────────────────────────────────────────────┐
│ Element          │ Value                                         │
│──────────────────────────────────────────────────────────────────│
│ title            │ NEET Kunoichi to Naze ka Dousei Hajimemashita │
│ episode          │ 01                                            │
│ episode_title    │ E02                                           │
│ video_resolution │ 1080p                                         │
│ audio_term       │ AAC                                           │
│ release_group    │ JapDub                                        │
│ source           │ Web-DL                                        │
│ file_extension   │ mkv                                           │
└──────────────────────────────────────────────────────────────────┘

9. ¯\(ツ)/¯

Output

>anitomy "e8cf4075112ea1a0.mp4"
┌───────────────────────────────────┐
│ Element        │ Value            │
│───────────────────────────────────│
│ title          │ e8cf4075112ea1a0 │
│ file_extension │ mp4              │
└───────────────────────────────────┘

10. Not sure what E1P means here, but it otherwise parses fine. (Edit: Ah yes, it's supposed to be EP1.)

Output

>anitomy "Ore wa Seikan Kokka no Akutoku Ryoushu! E1P [1080p][AAC][JapDub][GerEngSub][Web-DL].mkv"
┌────────────────────────────────────────────────────────────────┐
│ Element          │ Value                                       │
│────────────────────────────────────────────────────────────────│
│ title            │ Ore wa Seikan Kokka no Akutoku Ryoushu! E1P │
│ video_resolution │ 1080p                                       │
│ audio_term       │ AAC                                         │
│ release_group    │ JapDub                                      │
│ source           │ Web-DL                                      │
│ file_extension   │ mkv                                         │
└────────────────────────────────────────────────────────────────┘

[FichteFoll]

Yeah, 6 and 9 (and 10) are trackma-specific. I didn't put too much thought into each individual exception when dumping them here.

Using the anitomy CLI does look promising, though. It's not the highest priority for trackma currently, but I might check out integrating it when I find the time and interest. I'm also curious about the parsing speed and overhead involved when doing many invocations repeatedly, but realistically matching the result with the user's list takes much more time anyway.

One possible speedup would be to support a stdio- and line-based operation mode where each line (LF- or null-separated) in stdin is treated as a file name to be parsed and then anitomy emits a single JSON line/object for each input line.

[erengy]

There's already a --stdin option, but no batch processing. Might look into that sometime.

In any case, just wanted to put it out there as an option, in case it's preferable over forking the Python library to avoid duplicated effort. I'll continue working on v2 in the meantime.

[z411]

Hi @erengy - it's nice to have you here.

I definitely feel like we could use the original Anitomy project instead as we have shared goals. I feel that in theory FFI should be faster than calling an external binary repeatedly (especially when scanning the library) but we should probably profile both options to get make a proper decision. pip can also just build the C++ library by itself when installing, which is probably better integrated than having to compile an external binary.

Kinda unrelated but this reminds me that most of the time spent scanning the library is actually used up by difflib. I'm not a fan of adding tons of dependencies but if we can speed up the scanning then finding a faster fuzzy matching library could be a necessary evil. A user once reported that his big library took an hour (!) to scan and it has concerned me ever since.

[FichteFoll]

Library scanning has been improved significantly through caching via #748, but it could definitely be sped up even further. But that is for another day.

Since UNIX systems generally come with a C(++) compiler, building a C(++) extension there is definitely easier than for Windows, but in a first interation we should probably go with an externally provided executable instead. And I would definitely prefer an stdio-based communication over FFI.