Frequent IOExceptions using MVC with Core CLR on Linux

[dougbu]

Functional tests on Linux often fail due to IOExceptions. I currently see 97 test failures in Test/Microsoft.AspNet.Mvc.FunctionalTests and 78 IOExceptions in the output.

The root cause may be an external issue but should start with confirming we're using PhysicalFilesWatchers efficiently (given the number of files in the repo). Or perhaps this is a known system issue we should work around when testing on Linux (similar to setting MONO_THREADS_PER_CPU).

Example stack traces (think this covers all 78 IOExceptions):

      System.IO.IOException : No space left on device
      Stack Trace:
           at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter)
           at System.IO.FileSystemWatcher.RunningInstance.SysCall[TArg1,TArg2](Func`4 sysCall, TArg1 arg1, TArg2 arg2)
           at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
           at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
...
           at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyEvents notifyFilters, CancellationToken cancellationToken)
           at System.IO.FileSystemWatcher.StartRaisingEvents()
           at Microsoft.AspNet.FileProviders.PhysicalFilesWatcher.CreateFileChangeTrigger(String filter)
           at Microsoft.AspNet.Mvc.Razor.Compilation.CompilerCache.GetOrAdd(String relativePath, Func`2 compile)
           at Microsoft.AspNet.Mvc.Razor.VirtualPathRazorPageFactory.CreateInstance(String relativePath)

      System.IO.IOException : No space left on device
      Stack Trace:
           at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter)
           at System.IO.FileSystemWatcher.RunningInstance.SysCall[TArg1,TArg2](Func`4 sysCall, TArg1 arg1, TArg2 arg2)
           at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
           at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
...
           at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyEvents notifyFilters, CancellationToken cancellationToken)
           at System.IO.FileSystemWatcher.StartRaisingEvents()
           at Microsoft.AspNet.FileProviders.PhysicalFilesWatcher.CreateFileChangeTrigger(String filter)
           at Microsoft.AspNet.Mvc.Razor.Directives.DefaultChunkTreeCache.GetOrAdd(String pagePath, Func`2 getChunkTree)
           at Microsoft.AspNet.Mvc.Razor.Directives.ChunkInheritanceUtility.GetInheritedChunkTrees(String pagePath)

      System.IO.IOException : Too many open files
      Stack Trace:
           at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter)
           at System.IO.FileSystemWatcher.StartRaisingEvents()
           at Microsoft.AspNet.FileProviders.PhysicalFilesWatcher.CreateFileChangeTrigger(String filter)
           at Microsoft.AspNet.Mvc.Razor.Compilation.CompilerCache.GetOrAdd(String relativePath, Func`2 compile)
           at Microsoft.AspNet.Mvc.Razor.VirtualPathRazorPageFactory.CreateInstance(String relativePath)

      System.IO.IOException : Too many open files
      Stack Trace:
           at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter)
           at System.IO.FileSystemWatcher.StartRaisingEvents()
           at Microsoft.AspNet.FileProviders.PhysicalFilesWatcher.CreateFileChangeTrigger(String filter)
           at Microsoft.AspNet.Mvc.Razor.Directives.DefaultChunkTreeCache.GetOrAdd(String pagePath, Func`2 getChunkTree)
           at Microsoft.AspNet.Mvc.Razor.Directives.ChunkInheritanceUtility.GetInheritedChunkTrees(String pagePath)

FYI I reconfigured the system to bump up the ulimit -n number from 1024 to 10240 (see this answer). This had minimal impact.

[dougbu]

Likely we could work around this issue by caching HostingEnvironment or PhysicalFileProvider instances in the functional test's TestHelper class. HostingEnvironment.WebRootFileProvider is the PhysicalFileProvider instance experiencing the above issues.

These problems should not occur in a real application because the HostingEnvironment is a singleton and maps 1:1 down to a FileSystemWatcher instance. We don't create FileSystemWatcher instances per trigger. (We also reuse triggers per pattern and therefore, as MVC uses the WebRootFileProvider, per *.cshtml file.)

---

Separately I'm trying to see what resource we're actually exhausting...

[rynowak]

We also tend to create a new 'server' per testcase. We could probably speed up these tests a few seconds by not doing that.

[dougbu]

If users do functional testing of MVC behaviours, they may benefit from the more-specific error messages @stephentoub added in dotnet/corefx#3060.

Testing a PR to avoid inotify limits in our functional tests...

[dougbu]

Details on the inotify limits

Creating lots of FileSystemWatcher instances hits limits stored in /proc files with the following defaults:

> cat /proc/sys/fs/inotify/max_user_instances
128
> cat /proc/sys/fs/inotify/max_user_watches
8192

These can be increased temporarily e.g.

sudo sh -c "echo 2048 > /proc/sys/fs/inotify/max_user_instances"

or permanently e.g. adding the following to /etc/security/limits.conf.

fs.inotify.max_user_instances=2048
fs.inotify.max_user_watches=131072

[dougbu]

d03a851