fixin for relocatable perls

[schwern]

@shadowcat-mst brought up that fixin be changed to accommodate relocatable perls.

His suggestion that '#!/usr/bin/env perl' be allowed is less than ideal. Programs installed with MakeMaker typically are associated with a module, possibly a particular version of a module. There's a good chance that "whatever perl comes first in your PATH" isn't going to be the right one. Alternatively, it requires careful management of what your current Perl is.

Matt, why don't you explain the problem some more please and let's see if we can figure out something?

[shadowcat-mst]

Given a perl built with relocatable inc ... mv-ing the complete install around works except for the part where the scripts you've installed against it still aren't relocatable, because the #! line is still pointing to wherever the perl was -before- you mv'ed it somewhere else.

So, yes, using #!/usr/bin/env requires management of what your current perl is ... but it at least makes it possible to get this right, and I think most -Drelocatableinc perls are built to be "local" perls - i.e. they're going to get cp'ed/mv'ed around a lot, and will already be being used by setting up $PATH

So I'll give you "less than ideal" but what we currently do is just plain broken for this particular use case (and in fact is a direct reason why shadowcat's only customer to use a relocatable perl just dumped it for a single shared perl per box and a bunch of local::lib directory trees).

Suggestions for closer-to-ideal solutions very much welcome, mind, I've been using '#!/usr/bin/env perl' in my own codebases for a loooong time now so I've met and hated plenty of failure modes of it :)

[schwern]

I just want to make sure we have the problem right. You want to be able to
take a complete Perl install, where all the binaries and libraries are under
one root, and move it to another location. Good to know, I thought this was
about individual module installs.

#!/usr/bin/env perl will work, but as noted has issues. So let's call that
a backup plan and brain storm a little to see if we can come up with something
better.

Here's a few questions which affect the possible solutions...

• How often does relocation happen? A lot? Just once?
• Is relocation planned from the beginning? (ie. is this something like the
  ActivePerl installer?)
• Who does the relocation? The user? An installer?

What I'm angling at is if this is an infrequent occurrence, like with an
installer, then having a script which does the move and changes #! to the new
location seems like the way to go.

OTOH if you want something that you can, for example, just copy off a USB
drive and go, you might want to let the user run anything.

How about this, right now there's no way to tell fixin what Perl to use. So
we let the user override. That needs two fixes...

1. Have fixin() take the Perl to use as an argument.
2. Provide a Makefile.PL/WriteMakefile argument to state which Perl to use.

This covers both the "install everything using /usr/bin/env perl" case and
also allows one to write a spider using fixin to change #!. I see this as all
win.

There's an issue with part 1 in that the fixin interface has no space in it to
accept more arguments. Rather than wedge it in, I think there should be a new
method with a new name, one that makes more sense like change_shebang(), and
turn fixin() into a wrapper around that.

[schwern]

Here's the wackier idea.

What if #! exec'd a program which then re-executed the program using the Perl in the root directory? And if it had permissions it could even rewrite the #! line so it didn't have to do that again next time. This is sort of like the "running under some shell" code, but now it's "running under the wrong perl". The code could be injected into the program just like "running under some shell" is.

The benefit is you can move around the Perl install like any other directory tree, run them with any Perl you like, and the programs will fix themselves! No careful PATH management or installer necessary.

There are, of course, a bunch of issues. Detecting we're running in the wrong Perl. Reliably finding the right Perl. Keeping the injected code down to a minimum. The oddity of programs that alter their source code on disk when run. Preserving command line arguments.

This would be in parallel with making fixin more flexible.

[bowman]

I found this ticket while trying to work through a similar issue (#!env perl in dev & #!/custom/perl in production package).

It looks like _fixin_replace_shebang might have intended that non-"perl" interpreters be usable, but the maybe_command call prevents it. maybe_command uses -x and requires an absolute path, but is given only the last part, eg. "env":

https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/blob/master/lib/ExtUtils/MM_Unix.pm#L1167

That probably doesn't help resolve anything, even if I've understood that code correctly.

However, perhaps changing $Config{startperl} at perl build time could be something relocation friendly?

# env is in the PATH...
$ perl -MExtUtils::MM_Unix -E '$U=ExtUtils::MM_Unix; $ExtUtils::MakeMaker::Verbose++;' \
   -E 'say "\n", $U->_fixin_replace_shebang(test=>"#!/usr/bin/env perl")'
Can't find env in PATH, test unchanged at /.../ExtUtils/MM_Unix.pm line 1194.

# custom $Config{startperl} in ./Config.pm
$ perl -I. -MConfig -E 'say for $INC{"Config.pm"}, $Config{startperl}'
Config.pm
#!./bin/perl

$ perl -I. -MExtUtils::MM_Unix -E '$U=ExtUtils::MM_Unix; $ExtUtils::MakeMaker::Verbose++;'\
    -E 'say "\n", $U->_fixin_replace_shebang(test=>"#!/usr/bin/perl")'
Changing sharpbang in test to ./bin/perl
#!./bin/perl

eval 'exec ./bin/perl  -S $0 ${1+"$@"}'
    if 0; # not running under some shell

[karenetheridge]

Just how wedded are we to the relocatable feature? It doesn't sound like it's used much, and we're seeing that supporting it is a headache. Is it reasonable to table the possibility of dropping this feature?

[miyagawa]

relocatable INC is a feature in perl, and this is not the right place to discuss the possibility of dropping it. perl5-porters is. http://search.cpan.org/~rjbs/perl-5.17.11/INSTALL#Relocatable_@INC

[schwern]

I agree with @miyagawa, it's not our place to decide what bits of Perl are and aren't supported. Though we can tell them when one is a PITA.

@karenetheridge Relocation is very important if you, for example, want to put Perl on a USB stick. The path to Perl will change from machine to machine. So while it may not get used directly by many users, it is used indirectly by many users.

[shadowcat-mst]

If I'm building for a local::lib that's going to go with a relocatable perl
then the trampoline approach isn't going to work.

So while I'm pretty happy with nuking /usr/bin/env perl by default for normal
perls, I really don't see that we have a viable plan A for relocatables.

I do agree that 'rewrite to /usr/bin/env always' is plan B. I just can't see
anything else being anything except a case of "this is systems. you appear
to be trying to be clever. please stop."

Do please prove me wrong if you can though, I'd love to see a plan A that
-does- work for everything.

[schwern]

@shadowcat-mst By "the trampoline approach" do you mean this? I agree it's a bit wacky.

I don't think we can have a plan A that works for every situation because relocatables and normal installs have mutually conflicting needs. That need cannot be decided by the dist author who writes the #! line, the author of Module::Foo should not have a say in what sort of install you have. That's the installer's decision.

Having the user set perl Makefile.PL RELOCATABLE=1 is a start. The user can tell us what situation they're in and MakeMaker can act appropriately. Relocatable Perl distributions can configure CPAN shells to do that automatically and/or set PERL_MM_OPT appropriately. This seems a much better state than things are in now, where some dists have #!/usr/bin/env perl and some don't.

It occurs to me that we might be able to divine intent by checking $Config{userelocatableinc} but that's not always indicative of it being a relocatable distribution. For example, I believe ActiveState sets it so its binary installer can put Perl where the user chooses. OTOH Windows doesn't honor the #! line. OT3H RELOCATABLE is potentially not just about the #! line.

PS I'm not 100% sure understand what local::lib brings to the relocatable party.

[shadowcat-mst]

PS I'm not 100% sure understand what local::lib brings to the relocatable party.

Nothing, except for extra complication - the relocatable perl binary finds
its lib dirs using (effectively) ../lib - but the same approach doesn't
work the other way around, because modules can be installed into a local::lib
but still intended for use with a relocatable perl.

So I was mentioning it as an example of why the easy(ish) of the 'clever'
options don't work - sorry for not managing to make that clear.

Matt S Trout - Shadowcat Systems - Perl consulting with a commit bit and a clue

http://shadowcat.co.uk/blog/matt-s-trout/ http://twitter.com/shadowcat_mst/

Email me now on mst (at) shadowcat.co.uk and let's chat about how our Catalyst
commercial support, training and consultancy packages could help your team.

[demerphq]

On 30 April 2013 02:14, Karen Etheridge notifications@github.com wrote:

Just how wedded are we to the relocatable feature? It doesn't sound like
it's used much, and we're seeing that supporting it is a headache. Is it
reasonable to table the possibility of dropping this feature?

I would consider it heavily used as the Win32 AS perls use it.

Yves

perl -Mre=debug -e "/just|another|perl|hacker/"

[jandubois]

On Tue, May 7, 2013 at 4:54 AM, Yves Orton notifications@github.com wrote:

On 30 April 2013 02:14, Karen Etheridge notifications@github.com wrote:

Just how wedded are we to the relocatable feature?

I would consider it heavily used as the Win32 AS perls use it.

I think you are mixing different features here. There is no shebang line
on Windows, so the whole fixin part doesn't matter there.

Perl on Win32 dynamically locates the lib and site/lib folders relative to
the perl5xx.dll (corresponds to libperl.so on Linux), so the installation
is somewhat "relocatable" by default. This is a core Perl feature on Win32
since 5.005 and not specific to ActivePerl. It does not provide fully
automated relocation support by itself; things like Config_heavy.pl still
needs updating.

Cheers,
-Jan

[bingos]

Referencing open RT ticket here: https://rt.cpan.org/Ticket/Display.html?id=72104