Nontemporal (streaming) stores

[jedbrown]

Nontemporal store instructions (like movntpd in AVX, associated with intrinsics like _mm256_stream_pd), which allow writes to bypass cache, offer an important boost for some memory bandwidth-limited operations. They typically have alignment requirements. It would be useful if such memory semantics could be exposed through stdsimd.

[Lokathor]

I've seen them in core::arch before, but never been able to get much detail about them.

Question: If misused, do they allow cache syncing issues, or any other form of UB? Or are they purely a hint?

[calebzulawski]

I'm not sure it has any effect on safety, since vectors aren't Sync, but I believe non-temporal stores can cause reads to return different results on other processors

Edit: whoops, not sure what I was thinking, of course they're sync

[thomcc]

Reads can always return different results on different processors (even if you store and load with SeqCst the writes are allowed to be in a store buffer for some amount of time, it's just that when the reads do become available, they have a consistent total order — note that RMW operations complicate what I just said, although it's still more or less true), but non-temporal stores and loads can cause the operations to not respect the temporal ordering usually used when cache lines are retired.

The traditional example of this is that the DEC Alpha didn't retire cache lines in temporal order, and as a result you could have thread 1 populate a struct and set an atomic variable to a pointer to that struct, but when thread 2 reads it, it sees data that predates the assignment, even though that implies some sort of time-travel of the load.

That is to say, non-temporal loads/stores are effectively a atomic ordering weaker than even Relaxed, although it's a bit more complex since it refers to an abstraction layer below atomics (e.g. it refers to the cache coherency protocol that's used to implement atomics on x86), and so it's somewhat a leaky abstraction if you were use it as an actual ordering.

since vectors aren't Sync

They should be?

[calebzulawski]

It seems to me that a NT store function itself should be safe. Using it in a synchronization primitive or similar would be incorrect.

[thomcc]

It's tricky since it would break the guarantees of Acquire/Release orderings, e.g. Mutex<T> would have a data race if T were written with a non-temporal store. I think the safety guarantee is that you perform the appropriate fencing before the load.

[calebzulawski]

That's true, I do think you need an sfence after your stores. It's worth noting that there is a bit of discussion on NT operations in #16 and possibly in Zulip as well. I think our discussions have leaned towards nothing about NT ops being unique to SIMD, however it is somewhere they are more frequently used.

[thomcc]

I think given that existing code relies on Ordering::* behaving as described for safety, and this code would have data races when non-temporal operations are introduced, either:

1. NT stores should be unsafe, or
2. Vectors need to be !Send and !Sync

My preference would strongly be 1, since non-temporal operations are comparatively rare.

[jedbrown]

Oh thanks, I'd missed that issue (and we can close this to consolidate).

The core::intrinsics::nontemporal_store docs say

Probably will never become stable.

If it can't be stabilized for scalars, presumably it also wouldn't be considered for stabilization with SIMD. (I don't know the rationale for "never".)

[Lokathor]

core::intrinsics are just presumed to never become stable by default, that the ability will become stable elsewhere within the standard library eventually.

[jedbrown]

Yeah, though other intrinsics tend not to say they'll never be stable, and those that have stabilized now link to their stable counterparts.

[Lokathor]

Well having the very concept never be stable in any form forever is just unacceptable.

[hsivonen]

I think given that existing code relies on Ordering::* behaving as described for safety, and this code would have data races when non-temporal operations are introduced, either:

1. NT stores should be unsafe, or

2. Vectors need to be !Send and !Sync

My preference would strongly be 1, since non-temporal operations are comparatively rare.

Indeed, it would be very sad if vectors became !Send.

[workingjubilee]

For all operations in the Rust language and standard library where the operation is both supported by the compiler and is usable in a non-SIMD context, I see little reason to preempt the rest of the Rust stdlib in offering an interface specifically to a vector variant, even if the vector variant would get more usage, because it adds another layer of complications. We would want the scalar variant to compare against, if nothing else.

[jcranmer]

The experience I have with nontemporal stores suggests the following:

• Nontemporal stores do not follow the regular memory model rules for ordering.
• LLVM has indicated that they don't really wish to support nontemporals in any rigorously cross-platform way (see http://lists.llvm.org/pipermail/llvm-dev/2020-April/141279.html), save for perhaps a "nontemporal pre/post fence" (which does not presently exist).
• I don't know anything about non-x86 platforms.
• The recommendation on x86 is universally "add SFENCE after the loop." Omitting the SFENCE is not considered safe.

A bare nontemporal store is incredibly unsafe in that it opens you up to data races that need to be cured by special instructions that are not normally inserted by compilers. It needs to be paired with fences (while Clang supports a generic nontemporal intrinsic, it does not support a generic nontemporal fence intrinsic) to be safe.

I'd propose that any safe access to nontemporal stores must go through an interface that will issue the necessary fence as appropriate, and arguably also prohibit read access. Maybe something like this:

struct NontemporalBuffer<'a, T>(&'a mut [T]);
impl <'a, T> NontemporalBuffer<'a, T> {
   fn store(&mut self, index: usize, value: T) {
     let addr = &mut self.0[index];
     unsafe { std::intrinsics::nontemporal_store(addr, value); }
   }
}
impl <'a, T> Drop for NontemporalBuffer<'a, T> {
  fn drop(&mut self) {
    // TODO: Also insure that this gets run on panics
    unsafe { std::arch:x86_64::_mm_sfence(); }
  }
}

Admittedly, this interface is suboptimal if you want to write out to multiple streams, as each stream would cause a barrier to be emitted. (Maybe adding another type for the fence could work?) But this is the minimum level of interface you'd want in order to support safe access to nontemporal instructions.

[workingjubilee]

That could work, but I somewhat dread the prospect of attempting so without more extensive LLVM support... in other words: "that works on x64, sure, but what about everything else?"

This is unfortunately pretty clearly out of scope for us until a Rust codegen backend at least makes an attempt at supporting nontemporals and/or nontemporal fencing in a cross-platform way and we can guarantee we aren't blowing something up in our face. Further discussion can be taken to https://internals.rust-lang.org/ or, er, I suppose https://llvm.discourse.group/ also?