Support for sending messages to Metron

[youngm]

Thoughts on support for sending data to metron at some point?

If you want to reference existing work there is this community project but it appears to be abandoned: https://github.com/cloudfoundry-community/snotel

We currently use snotel to send messages to application log streams in our service brokers. We love the functionality but would like a more official regularly updated solution.

[nebhale]

I've never seen official documentation for how to send messages to Metron. I assumed this was intentional, but if you've got some, let me know and we'll look at it.

[youngm]

@nebhale I went over to the Loggregator channel and asked. Here is the answer I got:

youngm [1:22 PM]
@blove or @josmith Is the api to get messages into loggregator via metron_agent intended for non-loggregator components to use and documented somewhere?

blove [1:31 PM]
@youngm Yes the metron_agent is intended for other bosh deployed components outside of loggregator. We recommend using the V2 ingress API which is a gRPC API.
That API is defined here https://github.com/cloudfoundry/loggregator-api
We recommend using the BatchSender which is defined in the ingress.proto under the v2 directory.
If you are using Go, we do have a client package that can be used as well https://github.com/cloudfoundry/go-loggregator

youngm [2:32 PM]
Thanks @blove Is the v1 protocol documented at all beyond the protobuf definitions?

blove [2:35 PM]
If you prefer to use the V1 protocol, which is protobuf over UDP then there is https://github.com/cloudfoundry/dropsonde-protocol and https://github.com/cloudfoundry/dropsonde. We recommend using the V2 api

youngm [2:43 PM]
Thanks @blove!

blove [2:43 PM]
Np, let us know if you have any other questions

youngm [2:44 PM]
@blove what is the current status of the v2 grpc based api for egress?

blove [2:46 PM]
It is working, however it is not accessible outside of the bosh deployment, we are currently investigating how we can expose the v2 egress to the outside world.
We have a couple components internally that are using it, cf-syslog-drain-release and another one we are working on called log-cache-release.

Looking at the sites blove referenced it appears the API is considered public. But, may not be documented super well. It seems they refer to their client library implementation for api details beyond Protobuf and/or grpc. Is that good enough? :)

[nebhale]

Everything about this is troublesome. The firehose only exposes V1 types, so no way to test V2 ingress? No obvious documentation about what endpoint you'd connect to. API documentation is heavily through implementation. Also, V1 isn't (exclusively) over UDP, as the current client shows. Really have a lot of confidence about this. We'll put it on the backlog.

[youngm]

@nebhale Yeah not great. :) But, Loggregator's current strategy is to push V2 ingress and let loggregator adapt to whatever Egress protocol. So, you should be able to develop, support, and integration test ingress independent of egress.

Because this is a new feature I think you could jump right on V2 for ingress and not worry about v1.

And as a side I'd love to see a live grpc example using reactor netty. :)

[youngm]

Further clarification from loggregator team:

jasonkeene [5 hours ago]
Just to clarify, if you send data into loggregator via v2 it will be accessible from v1 and v2 egress. There is no reason to use v1 into metron agent.

jasonkeene [5 hours ago]
It is there for legacy writers that haven't migrated.

[youngm]

Here is an example of someone who could benefit from java-client supporting ingress. Interesting that the statsd injector may not be good enough for some use cases: cloudfoundry/statsd-injector#4