Packaging several py_binaries and runfiles with rules_pkg

[sashok-flex]

I have several py_binaries I want to package in a tarball. They each have runfiles, which have a lot of duplicate python dependencies (numpy, etc.). In the bazel cache, this is fine, since it's all symlinks. But when I attempt to use pkg_tar on these targets, the runfiles all get duplicated, which massively increases the size of the tarball.

What is the correct way to package several py_binaries with runfiles? Right now, I'm forced to hack a site-packages approach by manually packaging all of the python dependencies into a lib folder, which seems undesirable.

[philsc]

The way I've solved this in the past (2+ years ago) is to create a thin sh_binary that has all the py_binary targets as data dependencies. The thin shell wrapper just invokes the corresponding py_binary after a runfiles lookup. Then you tar up the sh_binary wrapper and it only has a single runfiles tree.

I admittedly have not tried this with a recent rules_python, but I suspect something like that should still work.

[philsc]

Something like this:

# load the runfiles library
...
# Delegate to the actual py_binary we care about.
bin=$1
shift
"$(rlocation $bin)" "$@"

Then when invoking the binary you run it roughly like so:

/app/wrapper repo/runfiles/path/to/py_binary arg1 arg2 arg3