feat: Prometheus metrics

[Kernald]

Please check existing knowledge before opening an issue

• I have checked the documentation
• I have searched through existing issues and discussions.

Describe the new functionality

There currently isn't a great way to monitor the status of a multi-scrobbler instance. The /api/health endpoint gives some good insight, but doesn't really follow any standard beyond returning a status code. It would be great having this exposed as Prometheus metrics, e.g. right now on my instance I have the following output for /api/health:

{
  "messages": [
    "Source ytmusic - kernald-ytm requires authentication.",
    "Source ytmusic - kernald-ytm is not polling."
  ]
}

Having metrics like:

multi_scrobbler_source_issues{ytmusic-kernald-ytm} 2

Would make it easy to actually monitor and firing alerts when something goes wrong.

Configuration Example

Will you help implement it?

• I am willing to contribute code towards completing this feature
• I am willing to help test preview builds for this feature

[FoxxMD]

@Kernald I've got basic prometheus metrics implemented like this

# HELP multiscrobbler_source_discovered Number of discovered plays for a Source
# TYPE multiscrobbler_source_discovered counter
multiscrobbler_source_discovered{name="myspot",type="spotify"} 50

# HELP multiscrobbler_client_queued Number of queued plays for a Client
# TYPE multiscrobbler_client_queued gauge
multiscrobbler_client_queued{name="koitodev",type="koito"} 21

# HELP multiscrobbler_client_deadletter Number of deadletter plays for a Client
# TYPE multiscrobbler_client_deadletter gauge

# HELP multiscrobbler_client_scrobbled Number of discovered plays for a Source
# TYPE multiscrobbler_client_scrobbled counter
multiscrobbler_client_scrobbled{name="koitodev",type="koito"} 4

# HELP multiscrobbler_client_issues Number of errors/issues with Client
# TYPE multiscrobbler_client_issues gauge
multiscrobbler_client_issues{name="koitodev",type="koito"} 0

# HELP multiscrobbler_source_issues Number of errors/issues with Source
# TYPE multiscrobbler_source_issues gauge
multiscrobbler_source_issues{name="myspot",type="spotify"} 0
multiscrobbler_source_issues{name="default",type="plex"} 0
multiscrobbler_source_issues{name="lztest",type="endpointlz"} 0
multiscrobbler_source_issues{name="lfmtest",type="endpointlfm"} 0
multiscrobbler_source_issues{name="mywebscrobbler",type="webscrobbler"} 0

Please try docker image

foxxmd/multi-scrobbler:pr-444

Prometheus usage is here: https://deploy-preview-444--multi-scrobbler.netlify.app/configuration/#prometheus

Let me know if this is sufficient for you or what you'd like to see changed.

[Kernald]

Amazing, it looks great! I'll give that a try later this weekend and update here :-)

[Kernald]

So one thing that seems to be an issue with Prometheus: the content-type is text/html; charset=utf-8, where it should be plain text. Prometheus refuses to scrape anything that's not plain-text by default.

Beyond that, I'll set up some alert rules from the metrics I get and update here, but from a quick glance it looks pretty good!

[Kernald]

So yeah after playing around today with the metrics for alerts, the existing metrics look great, thanks for this! Just the content-type to sort out and from a user perspective, it's ready to go :-)

[FoxxMD]

@Kernald I've updated the content-type. Please pull the latest pr-444 image and see if that fixes the issue, thanks

[Kernald]

It does, thanks! 🚀