Generate messages in downstream builds

[mjcarroll]

🎉 New feature

Expose message generation functionality to downstream packages and remove message generation from the msgs package.

Summary

Background

Up until now, we have:

• Distributed proto files - each describing a gz-msgs message
• Generated .cc and .hh files in the gz-msgs package
• The cc and hh files are compiled into a gz-msgs library that is also distributed in the binary package.

The issue with this approach is that it means that each change to a proto message will cause the ABI to break, which is inconvenient. We have multiple issues addressing this in

• Autogenerated protobuf code as public API #113
• Document policy for message modification #45
• Expose message generation framework #36

New Approach

This new approach will now distribute the proto files, but will no longer distribute a binary library that downstream users will link against. Instead, downstream users can generate the messages that they need themselves or add new message definitions in their own packages.

A summary of the changes required to make this happen:

• Expose the mechanisms we use for message generation
  • This requires installing the protoc generator as well as our python script.
  • Add cmake functionality to let downstream users discover available messages and retrieve the installation path of the generator and python script (handled via cmake extras)
• Create a core library that has the gz-msgs functionality minus the generated code.
• Create a compiled library that maintains the CLI functionality.
• Make conversions a header-only operation, so that the core library doesn't need to depend on the generated messages.

Downstream changes required:

• gz-fuel-tools Use updated message generate pipeline gz-fuel-tools#354
• gz-gui: Update gz-gui to use new message generation pipeline gz-gui#541
• gz-sim: Update gz-sim to use new message generation pipeline gz-sim#2000
• gz-sensors: Update gz-sensors to use new message generation pipeline gz-sensors#349
• gz-launch: Update gz-launch to use new message generation pipeline gz-launch#219
• gz-transport: Use the message generation framework gz-transport#398

Checklist

• Signed all commits for DCO
• Added tests
• Added example and/or tutorial
• Updated documentation (as needed)
• Updated migration guide (as needed)

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

[caguero]

Just checking, we can't compile the whole harmonic yet with only this PR, right?

[caguero]

Could you help me a bit with the expectations after this PR? What can I do and what can't I do yet (if any)?

[caguero]

Does this PR require https://github.com/gazebosim/gz-cmake/pull/345/files?

[mjcarroll]

Just checking, we can't compile the whole harmonic yet with only this PR, right?

Correct, I am working on the corresponding downstream modifications.

Could you help me a bit with the expectations after this PR? What can I do and what can't I do yet (if any)?

I still owe some documentation here. The general expectation is that gz-msgs will only install proto files as part of the debians, rather than generated headers and libraries. Downstream libraries will generate the cc/hh files from protos as part of their build process.

We should expect that with this PR, you should be able to add messages to your project, as demonstrated in this example: https://github.com/mjcarroll/mjcarroll-custom-msgs

This will also provide users a way to define custom messages downstream with simple cmake macros.

Does this PR require https://github.com/gazebosim/gz-cmake/pull/345/files?

Correct, it was just merged, and I'm going to cut a release shortly.

[mjcarroll]

@osrf-jenkins retest this please

[caguero]

I'm compiling Harmonic with the understanding that it won't successfully compile all libraries. After running colcon I see generated .pb.h files under ./build/gz-msgs10/gz_msgs_gen/gz/msgs. Is that expected?

[caguero]

I also see these two warning messages:

--- stderr: gz-msgs10                                                                                               
warning: tag INPUT: input source '/home/caguero/harmonic_ws/build/gz-msgs10/include/gz/msgs/details' does not exist
warning: tag INPUT: input source '/home/caguero/harmonic_ws/build/gz-msgs10/include/gz/msgs/details' does not exist

[mjcarroll]

I'm compiling Harmonic with the understanding that it won't successfully compile all libraries. After running colcon I see generated .pb.h files under ./build/gz-msgs10/gz_msgs_gen/gz/msgs. Is that expected?

Yes, this is correct. They are being generated to run the tests.

[mjcarroll]

@caguero I have opened PRs for all downstream libraries and added examples/READMEs.

[iche033]

Update Migradtion.md guide?

[codecov]

Codecov Report

Merging #339 (3944df9) into main (3c62ef7) will increase coverage by 1.46%.
The diff coverage is 97.85%.

❗ Current head 3944df9 differs from pull request most recent head a775cb5. Consider uploading reports for the commit a775cb5 to get more accurate results

@@            Coverage Diff             @@
##             main     #339      +/-   ##
==========================================
+ Coverage   95.43%   96.89%   +1.46%     
==========================================
  Files          10       28      +18     
  Lines        1030     1064      +34     
==========================================
+ Hits          983     1031      +48     
+ Misses         47       33      -14     

Impacted Files	Coverage Δ
core/generator/Generator.hh	100.00% <ø> (ø)
core/generator/generator_main.cc	100.00% <ø> (ø)
...re/include/gz/msgs/detail/PointCloudPackedUtils.hh	94.80% <ø> (ø)
core/include/gz/msgs/MessageFactory.hh	80.00% <80.00%> (ø)
core/src/DynamicFactory.cc	82.00% <82.00%> (ø)
...re/include/gz/msgs/convert/SphericalCoordinates.hh	86.95% <86.95%> (ø)
core/src/Factory.cc	87.50% <87.50%> (ø)
core/src/MessageFactory.cc	96.87% <96.87%> (ø)
compiled/src/gz.cc	90.90% <100.00%> (ø)
core/generator/Generator.cc	91.93% <100.00%> (ø)
... and 18 more

[mjcarroll]

Okay, all feedback addressed and rebased to get DCO happy. I think we are good here, and then a prerelease?

[scpeters]

given the other issues I've been seeing with the /usr/lib/<arch> folder in gazebo-release/gz-msgs10-release#3 and gazebosim/gz-cmake#360, I'm a bit suspicious that this hard-coded ../../.. is wrong