Add flags to export metrics and failed wf histories

[THardy98]

What was changed

Add flags to export scenario metrics, export worker metrics, and export failed histories. Exported artifacts are in JSON format. Added options are:

• --export-scenario-metrics
• --export-failed-histories
• --worker-export-metrics

Currently, the client-side metrics (scenario metrics and export failed histories) are only implemented for throughput stress.
The worker option is only implemented for the Go worker.

Options will no-op on different configurations

Why?

Provides ability to persist outcomes/insights from Omes runs.

1. Closes

2. How was this tested:

go run ./cmd run-scenario-with-worker \
--language go \
--scenario throughput_stress \
--duration 10m \
--export-scenario-metrics \
--export-failed-histories \
--worker-export-metrics

and

go run ./cmd run-scenario-with-worker \
--language go \
--scenario throughput_stress \
--duration 10m \
--export-scenario-metrics \
--export-failed-histories \
--run-id test1

with

go run ./cmd run-worker \
--language go \
--run-id test1 \
--worker-export-metrics

3. Any docs updates needed?
   No

[Sushisource]

This whole thing feels like more work than necessary. Let's just use Prometheus to record all this and then we can snapshot the DB when it's done. Maybe another option would work too, but let's not write our own custom metrics format and collection

[THardy98]

Let's just use Prometheus to record all this and then we can snapshot the DB when it's done

Could you clarify a bit here?

AFAICT, we don't spin up a prometheus server to scrape/store our metrics. We just expose an endpoint for prometheus metrics to be scraped from. So there is no collection currently, except in-memory on the client.

These metrics are collected on the worker, and it isn't immediately obvious to me how we'd want to send worker metrics over to the client, to expose on its /metrics endpoint (if we want to do that at all), in tandem with running a prometheus server

So I opted to export this as a file

Maybe another option would work too, but let's not write our own custom metrics format

Yeah, fair. I can replace the current resource tracking with prometheus instrumentation.

[Sushisource]

What I mean is that we can just run a local Prom instance in-lieu of running inside actual infra. If/when we run inside full-on infra-provisioned stuff we can just use a real Prom service, but if we don't have one it's very easy to just spin one up locally and export the stuff from that.

We don't need to send metrics from the worker to the client. Both sides can just expose prom metrics for scraping independently and that's fine.

[THardy98]

Closing in favor of splitting the PRs, given that there will be a notable difference between the metrics in this PR and the prom PR:
export history failures PR
will add prom when up