Avoid file cache trashing on Linux with mmapfs by using madvise() ?

[micoq]

With mmapfs, search queries load more data than necessary in the page cache. By default, every memory mapping done with mmap() (and FileChannel.map() in Java) on Linux is expected to be read (almost) sequentially. However, when a search request is done, the inverted index and other data structures seem to be read randomly so the system loads extra memory pages before and after the necessary page.
This results in a lot of I/O from the storage to warm up the file cache. In addition the cache is filling with unnecessary data and can evict more quickly the hot pages, slowing the next requests.
The problem is more visible with big indices (~1TB in our case).

To avoid this, Linux provides the madvise() syscall to change the prefetching behavior of memory maps. You can tell the system to avoid loading pages by using this with the flag MADV_RANDOM.
Unfortunately, Java doesn't use this syscall. Lucene provides a native library to do this : org.apache.lucene.store.NativePosixUtil but it doesn't seem to be used.

To illustrate this, I made some tests on readonly indices (~60GB) with a batch of search requests (bool requests on some fields with size=0, just document count). Each index have been optimized with _forcemerge:

                            Warm                  Cold
                       madv   mmap    nio   madv   mmap   nio

query  1               8276   9100   9422  13967  10487 10769
query  2                  9     10      9     95   1031    28
query  3                403    774    753   1019   1267   839
query  4                428    852    739    702   1025   857
query  5               4003   5591   5580   7970   6778  5947
query  6               1608   2237   2567   2611   2511  2594
query  7               5154   7193   7476   7890   7204  7943
query  8                438    705    707   1110   1211   793
query  9               2824   3922   4377   4143   4400  4237
query 10               2313   3235   3073   3086   3262  3471
average                2545   3361   3470   4259   3917  3747

consumed cache (Mio)      -      -      -   1607   7659  4687
storage I/O (Mio/s)       0      0      0    ~30   ~250  ~150

Each column represents a single test and results are in ms:

• "cold" is made after a fresh startup and empty caches (echo 3 > /proc/sys/vm/drop_caches)
• "warm" is the same test made right after the first one

The query cache and the request cache have been disabled.
The storage is made of spinning disks.
Elasticsearch version: 5.5.0.
OS: RHEL 7.3

You can see mmapfs is consuming more cache and IO than niofs.

In the madv column, I patched Lucene (MMapDirectory) to execute madvise(MADV_RANDOM) on each mapped file. This further improve the file cache and I/O consumption. In addition, the search are faster on warmed data.
To do this, I just add a single line in MMapDirectory.java:

final ByteBuffer[] map(String resourceDescription, FileChannel fc, long offset, long length) throws IOException {
...
  try {
    buffer = fc.map(MapMode.READ_ONLY, offset + bufferStart, bufSize);
    NativePosixUtil.madvise(buffer,NativePosixUtil.RANDOM); // here !
  } catch (IOException ioe) {
    throw convertMapFailedIOException(ioe, resourceDescription, bufSize);
  }
...
  return buffers;
}

Then I compile the shared native library libNativePosixUtil.so (with Lucene sources):

cd lucene-6.6.0/misc
ant build-native-unix

And finally, starts Elasticsearch with -Djava.library.path=/.../lucene-6.6.0/build/native/NativePosixUtil.so in jvm.options.

I didn't know if this solution can be applied in all cases and I didn't test all the cases (replication, merging, other queries...) but it could explain why mmapfs badly perform on large setups for searching. Some users reported a similar behavior like here, here or here.

I didn't know if there is a similar problem on Windows since it's memory management is different.

[jasontedor]

I think this should be opened as a Lucene issue?

[s1monw]

Lucene already has a directory for this but it requires native code etc. You can go and use it, adding a custom directory to ES with a plugin is pretty straight forward. Since Java 10 will add the ability to add O_DIRECT to streams / channels I think I'd want to wait for this and add it as an optional thing we can use if you run on Java 10 (which will come early next year I assume).

I hope this makes sense @micoq I will close this for now. we can still reopen if needed. Please feel free to continue the discussion here.

[micoq]

Actually, I didn't tested NativeUnixDirectory yet.
O_DIRECT could helps to reduce the cache trashing while merging/writing segments (and maybe for the translog or the shards restoration ?). It's a good idea to integrate this directly into the JVM.

In my case, the bottleneck was especially about the read operations while searching documents. I wanted to know why mmapfs performed badly on large setups contrary to niofs (since mmapfs is theoretically better because it avoid extra copies into an user space buffer).

[s1monw]

In my case, the bottleneck was especially about the read operations while searching documents. I wanted to know why mmapfs performed badly on large setups contrary to niofs (since mmapfs is theoretically better because it avoid extra copies into an user space buffer).

that is truly interesting. It almost seems that with madvice the OS is a bit more diligent with mapping memory into the cache. So this is all pure guessing but in default mode mmap will do initiate quite a bit of read-ahead while in MADV_RANDOM mode it won't do any readahead. In the NIO case I guess the readahead overhead amortizes the extra copy to a userspace buffer, that the OS needs to do in the NIO case which is actually surprising since there is quite a bit of an overhead when doing normal file IO compared to mmaps. The latter has literally no syscalls involved (ideally) when reading from it. Also seeks are basically pointer manipulations. I wonder if the reads from NIO if indices / files are very large are faster due to the better usage of FS caches and amortized or rather prevented read-aheads.

This is quite an interesting place to do some research. I am convinced we won't ship any native code in core by default but there might be room for a plugin here.

[s1monw]

@micoq can you tell how much memory your machine has and how much of it you are giving to elasticsearch when you run these tests?

[micoq]

@s1monw Sure.
This machine has 64 GB of memory (swap is disabled).

Elasticsearch is configured with a heap of 24GB:

jvm.options
-Xms24576m
-Xmx24576m

Compressed pointers are enabled:
[2017-12-18T18:05:47,129][INFO ][o.e.e.NodeEnvironment ] [cLvT3QE] heap size [24gb], compressed ordinary object pointers [true]

[micoq]

I didn't do the test with NIO but if you try to map a big file and touch a single byte every 2Mbytes, Linux will load the entire 2MB chunk from the disk. Here is a quick and dirty example to illustrate this (do a drop caches before executing it):

public static void main(String[] args) throws IOException {
    MMapDirectory dir = new MMapDirectory(Paths.get("/home/me/test"));
    dir.setPreload(false); // just to be sure
    IOContext ctx = new IOContext();
    IndexInput in = dir.openInput("bigfile.dat", ctx);
    long pos = 0;
    while(true) {
	    in.seek(pos);
	    //pos += 2 << 11; // 4 MB/s (a byte every single page or 4096 bytes)
	    //pos += 2 << 12; // 8 MB/s (a byte every 2 pages or 8192 bytes)
	    //pos += 2 << 13; // 16 MB/s
	    //pos += 2 << 14; // 32 MB/s
	    //pos += 2 << 15;  // 64 MB/s
	    //pos += 2 << 16;  // 128 MB/s
	    //pos += 2 << 17;  // 256 MB/s
	    //pos += 2 << 18;  // 512 MB/s (this is a theoretical value for 1ms delay)
	    //pos += 2 << 19;  // 1024 MB/s
	    pos += 2 << 20;  // 2048 MB/s
            //pos += 2 << 21; // 4096 MB/s
            //pos += 2 << 22; // 4096 MB/s (the speed doesn't increase after this)
	    try {
		    Thread.sleep(1); // set this to 10 or 100 if the storage doesn't keeps up
	    } catch (InterruptedException e) {}
	    in.readByte();
    }
}

Note that on OpenJDK 8 and Linux, if you preload the file with dir.setPreload(true) (the load() method from MappedByteBuffer: https://docs.oracle.com/javase/8/docs/api/java/nio/MappedByteBuffer.html), the JVM will execute madvise(MADV_WILLNEED) on the whole mapping (it's the only case where Java seems to use madvise()). On Windows, the JVM just touch all the pages by reading a single value on every page (like the example above with the first line uncommented). (MappedByteBuffer.c in the JDK source)

madvise(MADV_RANDOM) effectively disables readaheads. Unfortunately, I didn't find any other way (i.e. without a native library) to change the kernel behavior.

[childe]

@micoq maybe you can use blockdev --setra to change the kernel behavior.

[micoq]

@childe Thank you but unfortunately this doesn't have any effect (I tried different values, the default was 8192 blocks in my case). The bottleneck came from the memory management read ahead which is different from the block device read ahead.

The memory management read ahead is only used for mapped files and the block device read ahead for all read operations on storage devices whether using niofs or mmapfs.

In fact, the 2MB read ahead limit I observed in my tests seems to be hard coded in the kernel:
https://github.com/torvalds/linux/blob/master/mm/readahead.c#L236