Run multiple Grafana Agent processes

[patmaddox]

Change description

• Permit parallel grafana agent downloader processes
• Configure grafana agent gRPC port
• Add -m option to generator to specify PromEx module name

This allows us to configure multiple PromEx processes within our own application.

What problem does this solve?

We want to record metrics and publish dashboards to multiple sources (e.g. one for ops team, and one for biz team).

Example usage

Configure multiple PromEx processes and add them each to the supervisor:

  config :my_app, Core.PromExBiz,
    ...
    grafana: [
      host: "http://localhost:3000",
      ...
      folder_name: "myapp-biz"
    ],
    grafana_agent: [
      working_directory: Path.expand(".grafana_biz"),
      config_opts: [
        metrics_server_path: "/metrics/biz",
        ...
        instance: "dev-biz",
        prometheus_url: "http://prometheus-biz:9090/api/v1/write",
        ...
        agent_port: 4040,
        grpc_port: 9040
      ]
    ]

  config :my_app, Core.PromExOps,
    ...
    grafana: [
      host: "http://localhost:3000",
      ...
      folder_name: "myapp-ops"
    ],
    grafana_agent: [
      working_directory: Path.expand(".grafana_ops"),
      config_opts: [
        metrics_server_path: "/metrics/ops",
        ...
        instance: "dev-ops",
        prometheus_url: "http://prometheus-ops:9090/api/v1/write",
        ...
        agent_port: 4141,
        grpc_port: 9041
      ]
    ]

Additional details and screenshots

Checklist

• I have added unit tests to cover my changes.
• I have added documentation to cover my changes.
• My changes have passed unit tests and have been tested E2E in an example project.

[patmaddox]

@akoutmos seeking initial feedback

Do you think it would be useful to add a How-To about running multiple agents? Wasn't quite ready to check the "I have added documentation box" after only documenting the new grpc_port config.

What do you think of adding a -m ModuleName to the generator, so we can do e.g. mix prom_ex.gen.config --datasource myapp-ops -m PromExOps? Right now the generated module is named PromEx and would need to be updated.

[akoutmos]

Just looked through the code. It looks good to me. While I don't think it is a very common use case to run multiple agents...some documentation would definitely be good. Emphasis on not double-using ports would also be nice :D.

I am also ok with adding the -m flag so long as PromEx is the default when it is not provided.

[patmaddox]

@akoutmos done!

[coveralls]

Coverage increased (+0.04%) to 79.258% when pulling dd0b996 on patmaddox:multi-agent into 611912c on akoutmos:master.

[akoutmos]

lgtm! Thanks!