Add option to convert nulls for BQ keys

[jamesmcm]

Adds the convert-nulls option to convert nulls to "null" for use as keys in BigQuery diffs. This is very useful for comparing output
datasets when null is an expected value/category for a string key e.g. in datasets where null country codes might be expected for example.

From the code it seems this is already the behaviour in Avro diffs, it is only BQ diffs that would fail previously.

[idreeskhan]

Thanks for the contribution! Would like a few changes. Should also look into why the tests are failing and fix that if possible.

[idreeskhan]

Should match the existing behaviour with Avro. https://github.com/spotify/ratatool/blob/master/ratatool-diffy/src/main/scala/com/spotify/ratatool/diffy/BigDiffy.scala#L465

[idreeskhan]

Probably can reuse most of the existing avro code

[jamesmcm]

Do you also want to warn here? I was just thinking in our case it was like a valid category, so we had hundreds of thousands of rows like that (thus wanting to make it a command line argument before).

[idreeskhan]

Yea I think it makes sense to warn here as well, I'd rather err on the side of being too noisy first and course correct later if it's a problem

[codecov]

Codecov Report

Merging #472 (bab0e87) into master (09f1654) will decrease coverage by 0.10%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master     #472      +/-   ##
==========================================
- Coverage   70.55%   70.44%   -0.11%     
==========================================
  Files          35       35              
  Lines        1440     1445       +5     
  Branches      114      122       +8     
==========================================
+ Hits         1016     1018       +2     
- Misses        424      427       +3     

Flag	Coverage Δ
ratatoolCli	3.17% <0.00%> (-0.02%)	⬇️
ratatoolCommon	?
ratatoolDiffy	30.82% <50.00%> (+0.03%)	⬆️
ratatoolExamples	18.79% <0.00%> (-0.08%)	⬇️
ratatoolSampling	62.01% <ø> (ø)
ratatoolScalacheck	81.81% <ø> (ø)
ratatoolShapeless	5.27% <0.00%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...in/scala/com/spotify/ratatool/diffy/BigDiffy.scala	55.11% <50.00%> (-0.45%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09f1654...bab0e87. Read the comment docs.

[jamesmcm]

Btw I had some issues with the formatting (and there are still some small differences vs. before - is there a scalafmt config for the repo?

And do you want me to squash the commits when it is ready to merge?

[idreeskhan]

We don't have a scalafmt config right now but you can probably re-use the one from scio in the meantime, we follow the same style guide.

Squashing is preferred

[jamesmcm]

Fixed, btw using the scalafmt config from the golden path:

version = "2.4.2"
maxColumn = 100

binPack.literalArgumentLists = true

continuationIndent {
  callSite = 2
  defnSite = 2
}

newlines {
  afterImplicitKWInVerticalMultiline = true
  beforeImplicitKWInVerticalMultiline = true
  sometimesBeforeColonInMethodReturnType = true
}

docstrings = JavaDoc

project.git = false

rewrite {
  rules = [PreferCurlyFors, RedundantBraces, RedundantParens, SortImports]
  redundantBraces.maxLines = 1
}

Resulted in a lot of changes though so for now I just fixed it manually.

[idreeskhan]

Yea that's alright, I can address that separately afterwards