Tracking issue for alloc_system/alloc_jemalloc

[alexcrichton]

Introduced in RFC 1183 these two crates lie underneath the facade with their tracking issue hooked up to #27783. I'd like to open a separate issue for just these two crates.

We have a request in Gecko to stabilize the ability to route allocations to specifically malloc (on all platforms). This includes Windows, where currently alloc_system routes to functions like HeapAlloc.

Essentially, one of these crates, or perhaps a new crate like them, may want to have an accelerated path to stabilization before allocators themselves. For example I wouldn't want to stabilize #![allocator] or #![needs_allocator], but we could stabilize the semantics of extern crate alloc_malloc redirects all allocations to malloc (for example).

Nominating for discussion!

cc @rust-lang/libs
cc @froydnj

[Aatch]

Whatever the behaviour of alloc_malloc is, it'll impact the behaviour of the other allocator traits, though this probably isn't a worry in practice, just need to make sure we don't stabilize ourselves into a corner.

I think a malloc allocator crate is useful and general enough that we could probably make it a compiler switch of some sort, anything that supports Rust should have a functioning malloc available.

[sfackler]

Just to be clear, the only thing that would be stabilized here would be the fact that a crate named alloc_malloc exists and causes allocation calls to hit malloc/realloc/free, and it would have no stable items?

[aturon]

cc @nikomatsakis @pnkfelix

I think it's reasonable to fast-track stabilization of some way of getting pure malloc-based allocations. I'm slightly uneasy with extern crate having these kind of behind-the-scenes effects, but I also don't immediately see a better mechanism.

[alexcrichton]

@Aatch, @sfackler

To be clear, I'm thinking we could stabilize only the name extern crate alloc_malloc and no other APIs. Semantically you would essentially be guaranteed that if you link to that crate then all allocations will be implemented via malloc and friends (standard C APIs, not Windows-specific ones where possible).

Also to be clear, it is not clear to me that this is actually what Gecko wants and that it would suffice for them.

@aturon

I'm slightly uneasy with extern crate having these kind of behind-the-scenes effects

Yeah we've always had a nice property that you can basically concatenate any crates together, but unfortunately allocators and panic runtimes erodes this down a little more :(

[nikomatsakis]

@aturon

I'm slightly uneasy with extern crate having these kind of behind-the-scenes effects, but I also don't immediately see a better mechanism.

It seems reasonable to have the idea of "global language services" that can be required and which can admit only one implementation. I wouldn't want this to be an expandable, user-defined set, since it's kind of anti-modular (unlike say a trait), but we do need SOME answer that permits someone to say "gimme memory" or "panic" without having to have the implementor in scope. Tying it to very special crates seems... ok to me. At least I too cannot directly think of an alternative.

I guess I could imagine that merely adding the crate as an extern crate isn't enough, the app or somebody has to "bless it" for it to serve in its special role as a provider of that mechanism. But I'm not convincced that's valuable and it requires us to think up how it gets declared.

In any case, total composability of crates isn't really possible. For example, we already have the notion of #[no_mangle] functions and so forth. That is more-or-less the same thing, no?

[sfackler]

To make things a bit more explicit, we could take a similar approach as to what we do now for plugins:

#![allocator(alloc_malloc)]

That approach would also be nice in that you could have one allocator crate that calls into another and e.g. adds some instrumentation or whatever.

[froydnj]

Also to be clear, it is not clear to me that this is actually what Gecko wants and that it would suffice for them.

The proposed changes here would work for us in the short term. We'd still probably write our own allocator when that API was stabilized so we could take advantage of posix_memalign etc. being available on all our platforms, but this would fill in the gaps nicely.

[alexcrichton]

Talking with @aturon offline, some thoughts we have are:

• Something like #![allocator] here seems plausible. This would likely want to be an RFC amendment with the semantics:
  • Final artifacts can mention #![allocator], and they will use that allocator (e.g. no rlibs)
  • We whitelist maybe only a handful of names for now. I'd probably go for system and malloc, where system is what alloc_system does today and malloc is "use malloc everywhere, including windows".
• We can likely fast-track this for stabilization, but it won't be available before 1.11 (August 10, 2016).
• We'll likely deprecate this once it's stable to write your own custom allocator as that's a strict superset of the functionality (and most of the rest of the Rust community probably doesn't want the malloc crate, just the system crate).
• Is this absolutely necessary for Gecko just yet? This should only matter once Rust has a significant memory footprint in Gecko in theory?

[sfackler]

To be clear, the thing to deprecated would be #[allocator(malloc)], not the entire #[allocator] API, right??

[alexcrichton]

Indeed!