Initial plugin architecture and support for custom targets

[Diggsey]

This is very rough around the edges, but it adds support for plugins to rustup. This commit focuses on "target" plugins, which are designed to make it extremely easy to cross-compile.

As an example, one can now run this one command, after installing rustup with the default settings:
cargo build --target android:armeabi

And it will compile for android!

Caveat: the actual Android NDK download is not yet automated, partly because it's very large and until we have resumable downloads, using the browser is preferable, and partly because automating it will require scraping the download page's html, and also having a mechanism to view and accept the license.

On the other hand, it will handle all the tricky toolchain set up and argument passing!

The above command is very magical, and breaks down into these steps:

• The rustup proxy detects use of a "magic" --target parameter (one containing a colon)
• Rustup looks for a plugin with the name target-android
• No plugin with that name exists, so rustup tries to install it, by asking cargo to install rustup-plugin-target-android
• Rustup tells the plugin to add the target "armeabi" if it doesn't already exist
• The target-android plugin tells rustup to install the normal arm-linux-androideabi target
• The target-android plugin locates the android NDK, and tells it to create a standalone toolchain within the plugin's directory
• The rustup proxy tells the plugin to run the command cargo build for the "armeabi" target (stripping out the "magic" --target android:armeabi)
• The plugin runs CARGO_TARGET_ARM_LINUX_ANDROIDEABI_LINKER="<path_to_ndk_gcc>" cargo build --target arm-linux-androideabi
• The project is built for android!

Magic targets can also be explicitly added via rustup target add <plugin>:<target_descriptor>.

With this command, additional arguments can also be passed to the cargo install used to install the plugin. For example:
rustup target add android:armeabi -- --path src/plugins/target-android
This will install the plugin from a location on disk.

Finally, there is a new subcommand for explicitly managing plugins:

rustup plugin install <name> [-- <cargo install arguments>]
rustup plugin uninstall <name>
rustup plugin list

On crates.io, plugins are named: rustup-plugin-<plugin_name>
"target" plugins have a name starting with target-, eg. target-android, or on crates.io, rustup-plugin-target-android
When writing a "smart" target, the target- prefix is omitted, eg. --target android:armeabi

[Diggsey]

There are some security implications with this approach: for example, typoing android could install a malicious package, and it may be surprising that a build command could install a plugin. There are a few possible things that could be done to mitigate this, in order of severity:

1. Ask for confirmation before installing a new plugin when it's happening implicitly (via target add or a build command, rather than through the plugin install subcommand)

2. Keep a whitelist of plugins which can be implicitly installed from crates.io (non-whitelisted plugins would have to be installed explicitly prior to adding the target or invoking a build command)

3. Never install plugins implicitly

[brson]

OK, I've finally sat down to think about this @Diggsey. I'm really sorry it's taken me so
long. The general mechanics look about like I'd expect, but I have a number of
designish questions so I haven't reviewed the code closely.

I'd like to do some brainstorming about the requirements here to make sure
there's nothing obvious I'm overlooking, but most of what follows is just
questions and observations about the proposed design.

Overriding the syntax of cargo's --target.

This means "target" has different meaning and syntax in different positions in
the Rust toolchain and depending on whether you happen to be using rustup.
rustc gets to control the naming of targets.

It seems to me this naming scheme is demanded by the generality of the plugin
architecture, but we could alternately map targets to plugins.

Plugin-private directories

From the code I can't quite tell the naming scheme of the private plugin
directories. It looks to me like they are directly in the existing "toolchains"
directory.

Are there chances of name collisions here? How do these directories get cleaned
up during plugin uninstall? How do they get cleaned up during rustup uninstall?

Maybe it makes sense to have .rustup/plugins/<plugin-name>/ as a private
work area for each plugin.

NDK upgrades

How are NDK upgrades handled? In the current incarnation if the user upgrades
the android NDK their standalone toolchain will not be representative of their
NDK. What should be done in this situation?

NDK discovery

Is the ANDROID_NDK environment variable standard?

Automatic NDK installation

Automatic NDK installation is the ultimate goal. I think it's right that the
android plugin first tries to discover one locally, and as a fallback can
install it automatically to its private directory.

How do you envision automatic installation working? It seems to me there is
going to need to be an interactive component, for click-through licenses at
least, but also just to ask permission to go do this big chunk of I/O.

The interactivity combined with automatic installation when a specific
--target is requested seems tricky. Users probably shouldn't ever be hit with
an interactive prompt for any cargo command. It's not something they or their
tools will expect.

Android API levels

This patch hard-codes API level 21. How can we generalize this to allow
arbitrary API levels?

Doing everything as part of the build command

This PR does all the downloading and configuration automatically. While this is
a worthy goal, even today rustup will not automatically install targets when
requested. Seems inconsistent.

While I have an inclination to do this, it's not obvious it's desirable: it
imposes an unexpected large delay to do massive amounts of I/O and networking;
it has the interactivity issues mentioned previously (is it ok to just go do all
that network I/O without confirmation? what if there are click through
licenses?).

Plugins

The plugin-oriented design is central here, imposing several constraints. Can
you say more about why plugins are the right approach? This is framed as
a general solution, not specifically for native toolchain management. What other
use cases do you anticipate?

The use of plugins seems to force the custom target naming. Since rustup can't
a-priori know that arm-linux-androideabi is handled by any specific target, the
user must identify the plugin, not the target. This is quite a price to pay
cognitively: today it is well established that Rust identifies targets by target
triples. This introduces a rustup-specific way to talk about targets but only
when extra native toolchain support is needed.

User has to wait for the plugin to build before they can proceed to configure
the native toolchain.

Plugin lifecycle management must be considered. How and when are plugins
upgraded? How can they be discovered?

[Diggsey]

It seems to me this naming scheme is demanded by the generality of the plugin
architecture, but we could alternately map targets to plugins.

Could you clarify what you mean by this? This design doesn't alter the existing space of targets at all, it simply extends it with a set of targets including ":" (which would never otherwise be part of a target).

The left hand side of this target (split on the ":") maps to a plugin, while the right hand side is an opaque string which the plugin understands (ie. the format of this part is entirely determined by the plugin, and has no relation to "normal" targets)

Plugin-private directories

Each toolchain has it's own "plugins" folder:
<multirust home>\toolchains\<toolchain>\plugins

Within that folder, each plugin has its own directory <plugin name>, eg. target-android. Since plugin names map to crates.io package names, there's no possibility for conflict. The plugin is free to use that directory to store whatever it wants.

Given that cargo install automatically installs binaries in a bin subdirectory, it means the full path to a plugin's binary is:
<multirust home>\toolchains\<toolchain>\plugins\<plugin-name>\bin\<plugin-name>.exe

I was originally going to go with a rustup-wide plugins folder, with plugins all compiled for the host platform. However, I realised there are a lot of advantages to doing it this way:

• Writing plugins is a lot easier. You can assume your cfg options will match those of the rust toolchain you're going to be working with. The "toolchain multiplexing" side of rustup need not be encoded into every target plugin.
• You can have different plugins installed in different toolchains, so you get a way to easily enable a plugin just for a specific project.

NDK upgrades

This is one area I hadn't considered. It's slightly helped by the fact that the NDK toolchains are namespaced by API level, so if you update the NDK and want to use new features, everything will still work (since you would have to up your API level, which would result in a fresh toolchain being generated, using the new NDK)

This will not handle bug-fixes to older API versions though, so we'll need to think about that.

NDK discovery

The ANDROID_NDK variable is not officially standardised, and is not set by the installer. However, it was used by several tutorials I found for cross compiling to android, including the servo instructions. I definitely think we can improve discovery, but this was mainly intended as a proof of concept, and given how target specific discovery will be, I don't see it affecting rustup's plugin interface significantly (although obviously it will have a big impact on the plugins themselves).

Android API levels

The API level is not hard-coded, it just defaults to 21 if left unspecified. It can be set as part of the plugin-defined target, and you can also use a custom STL (these are the only two configuration options supported by the NDK when generating a native toolchain), eg. --target android:armeabi,api=18,stl=gnustl

It's foreseeable that for some targets, there could be too many configuration options for them to be contained within a target string to be specified on every build. However, I found that the existing plugin architecture has a solution:

We define a new target plugin, eg. target-file which takes a user-defined target name as its target string, eg. --target file:my-special-target. It then looks up "my-special-target" in a TOML config file, (possibly by extending Cargo.toml) which maps the name to a full specification of that target.

That full specification would include a plugin name and arbitrary other options, and those options would be serialised to a string to be passed as the plugins target string, eg.

[my-special-target]
plugin = "android"
arch = "armeabi"
args = { "api": 18 }

Translates to android:armeabi,api=18 (actual serialisation scheme could be totally different)

Doing everything as part of the build command

Yep, I'm not set on doing everything as part of build, it was more of a gimmick to show that we could do if we wanted. However, we could suggest what command to run to install the target if it's not installed, so that the user doesn't need to look elsewhere to figure out how to cross compile. (Same goes for "normal" targets)

The plugin-oriented design is central here, imposing several constraints. Can
you say more about why plugins are the right approach?

If you want to cross compile today, with rust or any other language, the first thing you'll do is try to find a tutorial online, since generally someone will have at least done something similar before. Now instead/as well as writing a tutorial, that person can actually publish their work on crates.io, making that target immediately discoverable and available to any rustup user. I tried very hard to make it as easy as humanly possible to make these target plugins, to encourage this.

Given how many potential targets there are, I don't see any possible other way this could work, than with a user-extensible system.

This is framed as a general solution, not specifically for native toolchain management. What other
use cases do you anticipate?

I've very much focused on "target" plugins for the moment since they are the obvious case, although they're not necessarily restricted to NDKs. It could be the case that they simply pull in supporting libraries, or change the compilation flags, etc.

As for other plugin types, I don't have any particularly compelling use-cases, but here are my thoughts:

• rustup CLI extensions. These would exactly mirror cargo-<command> crates and would just add a subcommand to rustup. It's not clear to me that even if we do want to support this, that it should go through this plugin system (ie. they don't really need any non-trivial support from rustup to function). Plus there's the fact that they'll generally sit outside the toolchain, or else you'd just use a cargo plugin.
• Toolchain modifiers. These might add the ability to install additional components from a 3rd party for example.

In any case, I felt that the cross-target use-case alone was sufficiently compelling to justify this kind of plugin system, and the only real concession it makes to potentially supporting other types in future is through the naming scheme.

The use of plugins seems to force the custom target naming. Since rustup can't a-priori know that arm-linux-androideabi is handled by any specific target, the user must identify the plugin, not the target.

True, but I think in general this is preferable: with target plugins we're moving into a very heterogeneous area, where even simple definitions such as "what linker's are available for this target" can be completely meaningless for some targets. This is the fundamental problem with targets: they don't actually give a complete specification! There are many, many ways to compile for any given target triple, (just look at the current RFC about static vs dynamic linking the CRT!) and referring to targets by a plugin name plus a plugin-defined specification string completely solves this issue, is very flexible, and I think actually reduces the cognitive load.

eg. It's much easier to write --target android:armeabi,api=18 and know that it will do the right thing, compared to:

1. Looking up the correct android target triple for ARM (they're not even consistently named across architectures...)
2. Figure out how to specify android-specific options, such as API level
3. Figure out if I should alter the compiler flags because of peculiarities of the android platform. (For example, maybe for some platforms all code must be position independent?)

Basically, there are literally millions of possible targets, but those targets can be split into a fairly small set of groups (for example, android targets), and generally we only actually want to think about what group its in.

Plugin lifecycle management must be considered. How and when are plugins
upgraded? How can they be discovered?

Yep, I haven't put to much thought into this, but I imagine we'd do it in a similar way to toolchains. (ie. have a plain rustup plugin update update all plugins, or be able to specify a single one to update).
To be fair though, cargo has got on perfectly fine with still no way to update a cargo installed binary, so I'm not too worried about this.

[brson]

Sorry it's again taken me so long to come back to this.

Could you clarify what you mean by this? This design doesn't alter the existing space of targets at all, it simply extends it with a set of targets including ":" (which would never otherwise be part of a target).

I think I clarified this in other comments, but what I was trying to say is there is no way here for --target=arm-linux-androideabi to invoke the right plugin - because the android support is in a plugin with a name unknown to rustup, the user must name the plugin; there is no option to just name the target and get the ndk configuration.

Each toolchain has it's own "plugins" folder:
\toolchains\plugins

I see. I missed that there was a 'plugins' subfolder here.

Writing plugins is a lot easier. You can assume your cfg options will match those of the rust toolchain you're going to be working with. The "toolchain multiplexing" side of rustup need not be encoded into every target plugin.

In what situations would the specifics of the toolchain need to be encoded in the plugin?

You can have different plugins installed in different toolchains, so you get a way to easily enable a plugin just for a specific project.

Toolchains aren't per project in general, though one could use them that way. In what situations would one want to install plugins per-project?

Is the standalone NDK created under the plugin directory?

The API level is not hard-coded, it just defaults to 21 if left unspecified. It can be set as part of the plugin-defined target, and you can also use a custom STL (these are the only two configuration options supported by the NDK when generating a native toolchain), eg. --target android:armeabi,api=18,stl=gnustl

Thanks. I see. Here both api level and gnustl are things that have to be configured when creating the standalone ndk?

Given how many potential targets there are, I don't see any possible other way this could work, than with a user-extensible system.

I've identified 10 cross-compilation scenarios that might be served by this feature (further below). I'm sure there are more, and the configurations are complex; but there are a few crucial ones (like android). I'm not sure there are a great number, and many will be variations, like "find the right gcc for target X on Debian".

True, but I think in general this is preferable: with target plugins we're moving into a very heterogeneous area, where even simple definitions such as "what linker's are available for this target" can be completely meaningless for some targets. This is the fundamental problem with targets: they don't actually give a complete specification! There are many, many ways to compile for any given target triple, (just look at the current RFC about static vs dynamic linking the CRT!) and referring to targets by a plugin name plus a plugin-defined specification string completely solves this issue, is very flexible, and I think actually reduces the cognitive load.

The RFC about static vs. dynamic linking though solves this problem upstream by adding a mechanism to toggle a feature with rustc and cargo. I see the android API and it's STL don't have an obvious upstream solution. Are there other cases where the toolchain needs to be configured in a way that doesn't make sense for rustc itself? It probably is useful to be able to talk about targets plus their configurations, and that includes things like CPU features. It's an interesting problem that seems to overlap some other things going on in Rust.

To help me clarify my thinking about this I finally got around to sketching out some requirements, below. TTYL.

general rustup native toolchain requirements

• discover native toolchain components in platform-specific ways
• configure cargo/rustc with the correct linker
• define a private area for native toolchains to maintain their state
• correctly maintain state of ndk components across rustup upgrades
• detect if the proper cross-compile tools are not available and give a decent error message
• don't override the linker if the user is already doing that in some way

specific requirements

• create the android standalone ndk
• work with the android ndk on linux/mac/win
• support android api levels
• support various android c++ configurations
• find and configure the correct linker for musl on linux
• find and configure the correct linker for iOS on mac?

possible requirements

• install native toolchain components in platform-specific ways
• do interactive installation where necessary, e.g. click-through licenses
• automatically install native toolchains, ala pkg
• manage the android sdk

unknowns

• Is it rustup's role to configure the C toolchain? Seems like gcc-rs is pretty
  good at that most in some cases, though some cases are complex, particularly
  with android, and any case where rustup might be managing the native toolchain
  itself. Maybe there's some division of labor where rustup is commicating to
  gcc-rs certain important information.
• Is this entirely about detecting/installing the correct linker and configuring
  the rustc linker?
• How will rustup interact with testing of cross-compile targets?

future cross-scenarios

• linux->android
• win->android
• mac->android
• mac->iOS
• linux->mac
• linux->iOS?
• linux->linux-musl
• linux->win-gnu
• linux->win-msvc
• linux->bsd
• linux->etc.

[Diggsey]

Going to close this, as it's quite out of date at this point.