Data Streams messages - support Avro & Protobuf formats

[piochelepiotr]

What does this PR do?

For users using Avro & Protobuf, this PR allows them to retrieve messages from Kafka via the Datadog UI:

Avro support is pretty straight forward: The Avro schema is in Json format.
For Protobuf, the schema is the descriptor file, and in order to pass it via remote configuration & as a configuration to the Kafka Consumer integration, it is base64 encoded on top of it.
For Protobuf, it is assumed that the first object in the pool is the parent schema.

Increased Datadog agent size

https://gitlab.ddbuild.io/DataDog/datadog-agent/-/jobs/1051663772

Image size on disk (uncompressed image size) 1029868244 is higher than the maximum allowed 1029491916 by the gate !

So assuming the limit was close to the size before, that's an increase of 350Kb

Motivation

Most Kafka users are using Avro & Protobuf formats. With support for both of these, most users should be covered.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[codecov]

Codecov Report

❌ Patch coverage is 86.95652% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.19%. Comparing base (7709840) to head (80ab003).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
activemq	?
cassandra	?
confluent_platform	?
hive	?
hivemq	?
hudi	?
ignite	?
jboss_wildfly	?
kafka	?
kafka_consumer	89.93% <86.95%> (+0.99%)	⬆️
presto	?
solr	?
tomcat	?
weblogic	?

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gitstevenpham]

The schema is in bytes here but I believe the remote config will have string representation. Will there be an issue with that?

[gitstevenpham]

Clarified offline

[gitstevenpham]

lgtm

[steveny91]

Looks good to me. Maybe in the future it might be worth it to split off into a separate module/file for easier maintenance.

[piochelepiotr]

Looks good to me. Maybe in the future it might be worth it to split off into a separate module/file for easier maintenance.

Can definitely do that! I will want to add payload filtering next. I will make sure to split the code up in modules when I work on that.