Error on metadata schema mismatch

[whd]

We had a pipeline issue (in stage) this week due to an errant metadata generation change:

#219
#220

Fortunately when we deploy these kinds of changes SRE is generally made aware, and well-formed parts of the metadata in this case explicitly blocked the pipeline from deploying without operator approval. However, in the normal case (since this wasn't a BQ schemas error), this change might have auto-deployed to production even though the stage decoder wound up in a broken (but running) state.

We can probably either have tests to verify metadata is valid or else have the application run json schema validation checks after generating schemas, refusing to push schemas if there are issues.

[jklukas]

We already import the python jsonschema module in this codebase. I think it would be reasonable to add some code to validate all generated schemas against the metaschema from the main branch of mozilla-pipeline-schemas. We can call that in the main bash script similar to how we do the validate-bigquery call.

[jklukas]

I recently wrote up a proposal for supporting full mozPipelineMetadata for Glean apps which would move metadata preparation to probe-scraper, so this concern would move there.