NO_EXIT_RUNTIME by default (#5878)

This makes us not exit the runtime by default. That means we don't emit code for atexits and other things that happen when the runtime shuts down, like flushing the stdio streams. This is beneficial for 2 reasons:

 * For #5794, this helps remove code. It avoids all the support for shutting down the runtime, emitting atexits, etc. It also enables more optimizations (the ctor evaller in wasm can do better without calls to atexit). This removes 3% of hello world's wasm size and 0.5% of its JS.
  * A saner default for the web. A program on the web that does anything asynchronous will not want the runtime to exit when main() exits, so we set this flag to 1 for many tests, which this PR lets us remove.

However, this is a breaking change. As already mentioned, the possible breakages are

 * printf("hello") will not console.log since there is no newline. Only when the streams are flushed would that be printed out. So this change would make us not emit that.
 * atexits do not run.

Both of those risks are mitigated in this PR: In ASSERTIONS mode, check if there is unflushed stream output, and explain what to do if so. Same if atexit is called.

This PR has a lot of test changes, some that simplify web tests - because the new default is better for the web - but others that add a param to a shell test - because the new default is less optimal in a shell environment. I think the risk here is lower than those shell tests indicate: we do test quite a lot of things in the shell, but just because it's convenient, not because that's what most users care about.

Also:

 * this PR found an unnoticed bug: FORCE_FILESYSTEM didn't actually do what the name suggests. I think we just never tested it properly with NO_EXIT_RUNTIME. Fixed in this PR.

 * emrun sets NO_EXIT_RUNTIME=0. it is a mode where we specifically want to get the exit code from the running program, as if it were a shell command, not a browser app

 * add faq entry, and mention the faq

 * fix an existing emterpreter-async bug: if we are unwinding the stack as we leave main(), then do not call exit, we are not exiting yet - code is yet to run later

* metadce is now more effective, update test

* faq entry on Module.* is not a function

* fix browser.test_emscripten_main_loop - the pthreads part needs the runtime to exit

Author: kripken

ChangeLog.markdown

@@ -9,6 +9,7 @@ Not all changes are documented here. In particular, new features, user-oriented
 
 Current Trunk
 -------------
+ - Breaking change: Change `NO_EXIT_RUNTIME` to 1 by default. This means that by default we don't include code to shut down the runtime, flush stdio streams, run atexits, etc., which is better for code size. When `ASSERTIONS` is on, we warn at runtime if there is text buffered in the streams that should be flushed, or atexits are used.
  - Meta-DCE for JS+wasm: remove unused code between JS+wasm more aggressively. This should not break valid code, but may break code that depended on unused code being kept around (like using a function from outside the emitted JS without exporting it - only exported things are guaranteed to be kept alive through optimization).
 
 v1.37.24: 12/13/2017

emcc.py

@@ -479,7 +479,9 @@ def filter_emscripten_options(argv):
       # The preprocessor define EMSCRIPTEN is deprecated. Don't pass it to code in strict mode. Code should use the define __EMSCRIPTEN__ instead.
       if not shared.Settings.STRICT:
         cmd += ['-DEMSCRIPTEN']
-    if use_js: cmd += ['-s', 'ERROR_ON_UNDEFINED_SYMBOLS=1'] # configure tests should fail when an undefined symbol exists
+    if use_js:
+      cmd += ['-s', 'ERROR_ON_UNDEFINED_SYMBOLS=1'] # configure tests should fail when an undefined symbol exists
+      cmd += ['-s', 'NO_EXIT_RUNTIME=0'] # configure tests want a more shell-like style, where we emit return codes on exit()
 
     logging.debug('just configuring: ' + ' '.join(cmd))
     if debug_configure: open(tempout, 'a').write('emcc, just configuring: ' + ' '.join(cmd) + '\n\n')
@@ -663,6 +665,8 @@ def detect_fixed_language_mode(args):
       if options.emrun:
         options.pre_js += open(shared.path_from_root('src', 'emrun_prejs.js')).read() + '\n'
         options.post_js += open(shared.path_from_root('src', 'emrun_postjs.js')).read() + '\n'
+        # emrun mode waits on program exit
+        shared.Settings.NO_EXIT_RUNTIME = 0
 
       if options.cpu_profiler:
         options.post_js += open(shared.path_from_root('src', 'cpuprofiler.js')).read() + '\n'
@@ -1072,9 +1076,12 @@ def check(input_file):
 
       if not shared.Settings.NO_FILESYSTEM and not shared.Settings.ONLY_MY_CODE:
         shared.Settings.EXPORTED_FUNCTIONS += ['___errno_location'] # so FS can report errno back to C
-        if not shared.Settings.NO_EXIT_RUNTIME:
-          shared.Settings.EXPORTED_FUNCTIONS += ['_fflush'] # to flush the streams on FS quit
-                                                            # TODO this forces 4 syscalls, maybe we should avoid it?
+        # to flush streams on FS exit, we need to be able to call fflush
+        # we only include it if the runtime is exitable, or when ASSERTIONS
+        # (ASSERTIONS will check that streams do not need to be flushed,
+        # helping people see when they should have disabled NO_EXIT_RUNTIME)
+        if not shared.Settings.NO_EXIT_RUNTIME or shared.Settings.ASSERTIONS:
+          shared.Settings.EXPORTED_FUNCTIONS += ['_fflush']
 
       if shared.Settings.USE_PTHREADS:
         if not any(s.startswith('PTHREAD_POOL_SIZE=') for s in settings_changes):
@@ -1209,10 +1216,6 @@ def check(input_file):
           if 'interpret' in shared.Settings.BINARYEN_METHOD:
             logging.warning('disabling EVAL_CTORS as the bundled interpreter confuses the ctor tool')
             shared.Settings.EVAL_CTORS = 0
-          else:
-            # for wasm, we really want no-exit-runtime, so that atexits don't stop us
-            if final_suffix in JS_CONTAINING_SUFFIXES and not shared.Settings.NO_EXIT_RUNTIME:
-              logging.warning('you should enable  -s NO_EXIT_RUNTIME=1  so that EVAL_CTORS can work at full efficiency (it gets rid of atexit calls which might disrupt EVAL_CTORS)')
 
       # memory growth does not work in dynamic linking, except for wasm
       if not shared.Settings.WASM and (shared.Settings.MAIN_MODULE or shared.Settings.SIDE_MODULE):

site/source/docs/api_reference/emscripten.h.rst

@@ -334,6 +334,8 @@ Functions
 	
 	The difference is that ``emscripten_force_exit`` will shut down the runtime even if you previously called :c:func:`emscripten_exit_with_live_runtime` or otherwise kept the runtime alive. In other words, this method gives you the option to completely shut down the runtime after it was kept alive beyond the completion of ``main()``.
 
+	Note that if ``NO_EXIT_RUNTIME`` is set (which it is by default) then the runtime cannot be shut down, as we do not include the code to do so. Build with ``-s NO_EXIT_RUNTIME=0`` if you want to be able to exit the runtime.
+
 	:param int status: The same as for the *libc* function `exit() <http://linux.die.net/man/3/exit>`_.
 
 .. c:function:: double emscripten_get_device_pixel_ratio(void)

site/source/docs/getting_started/FAQ.rst

@@ -230,7 +230,7 @@ Another option is to define an ``onRuntimeInitialized`` function,
 
   Module['onRuntimeInitialized'] = function() { ... };
 
-That method will be called when the runtime is ready and it is ok for you to call compiled code. In practice, that is exactly the same time at which ``main()`` would be called, so ``onRuntimeInitialized`` doesn't let you do anything new, but it can be convenient in some cases - for example, if you use ``onRuntimeInitialized`` and don't define a ``main()`` function, then the runtime will not be shut down after ``main()`` exits, and you can keep calling compiled methods (you can also have a ``main()`` and build with ``-s NO_EXIT_RUNTIME=1`` to keep the runtime from being shut down). Thus, for libraries, ``onRuntimeInitialized`` can be convenient.
+That method will be called when the runtime is ready and it is ok for you to call compiled code. In practice, that is exactly the same time at which ``main()`` would be called, so ``onRuntimeInitialized`` doesn't let you do anything new, but you can set it from JavaScript at runtime in a flexible way.
 
 Here is an example of how to use it:
 
@@ -248,6 +248,27 @@ Here is an example of how to use it:
 The crucial thing is that ``Module`` exists, and has the property ``onRuntimeInitialized``, before the script containing emscripten output (``my_project.js`` in this example) is loaded.
 
 
+.. _faq-NO_EXIT_RUNTIME:
+
+What does "exiting the runtime" mean? Why don't ``atexit()s`` run?
+==================================================================
+
+(You may need this answer if you see an error saying something like ``atexit() called, but NO_EXIT_RUNTIME`` or ``stdio streams had content in them that was not flushed. you should set NO_EXIT_RUNTIME to 0``.)
+
+By default Emscripten sets ``NO_EXIT_RUNTIME=1``, which means that we don't include code to shut down the runtime. That means that when ``main()`` exits, we don't flush the stdio streams, or call the destructors of global C++ objects, or call ``atexit`` callbacks. This lets us emit smaller code by default, and is normally what you want on the web: even though ``main()`` exited, you may have something asynchronous happening later that you want to execute.
+
+In some cases, though, you may want a more "commandline" experience, where we do shut down the runtime when ``main()`` exits. You can build with ``-s NO_EXIT_RUNTIME=0``, and then we will call ``atexits`` and so forth. When you build with ``ASSERTIONS``, you should get a warning when you need this. For example, if your program prints something without a newline,
+
+::
+
+  #include <stdio.h>
+
+  int main() {
+    printf("hello"); // note no newline
+  }
+
+If we don't shut down the runtime and flush the stdio streams, "hello" won't be printed. In an ``ASSERTIONS`` build you'll get a notification saying ``stdio streams had content in them that was not flushed. you should set NO_EXIT_RUNTIME to 0``.
+
 .. _faq-dead-code-elimination:
 
 Why do functions in my C/C++ source code vanish when I compile to JavaScript, and/or I get ``No functions to process``?
@@ -324,6 +345,14 @@ Emscripten by default does *not* give fatal errors on undefined symbols, so you
 
 Aside from just forgetting to link in a necessary object file, one possible cause for this error is inline functions in headers. If you have a header with ``inline int my_func() { .. }`` then *Clang* may not actually inline the function (since inline is just a hint), and also not generate code for it (since it's in a header). The result is that the generated bitcode and JavaScript will not have that function implemented. One solution is to add ``static`` to the function declaration, which forces code to be generated in the object file: ``static inline int my_func() { .. }``.
 
+Why do I get ``TypeError: Module.someThing is not a function``?
+===============================================================
+
+The ``Module`` object will contain exported methods. For something to appear there, you should add it to ``EXPORTED_FUNCTIONS`` for compiled code, or ``EXTRA_EXPORTED_RUNTIME_METHODS`` for a runtime method (like ``getValue``).
+
+.. note:: You can use runtime methods directly, without exporting them, if the compiler can see them used. For example, you can use ``getValue`` in ``EM_ASM`` code, or a ``--pre-js``, by calling it directly. The optimizer will not remove that JS runtime method because it sees it is used. You only need to use ``Module.getValue`` if you want to call that method from outside the JS code the compiler can see, and then you need to export it.
+
+.. note:: Emscripten used to export many runtime methods by default. This increased code size, and for that reason we've changed that default. If you depend on something that used to be exported, you should see a warning pointing you to the solution, in an unoptimized build, or a build with ``ASSERTIONS`` enabled, which we hope will minimize any annoyance. See ``Changelog.markdown`` for details.
 
 Why do I get an odd python error complaining about libcxx.bc or libcxxabi.bc?
 =============================================================================

site/source/docs/optimizing/Optimizing-Code.rst

@@ -38,18 +38,6 @@ Advanced compiler settings
 
 There are several flags you can :ref:`pass to the compiler <emcc-s-option-value>` to affect code generation, which will also affect performance — for example :ref:`DISABLE_EXCEPTION_CATCHING <optimizing-code-exception-catching>`. These are documented in `src/settings.js <https://github.com/kripken/emscripten/blob/master/src/settings.js>`_. Some of these will be directly affected by the optimization settings (you can find out which ones by searching for ``apply_opt_level`` in `tools/shared.py <https://github.com/kripken/emscripten/blob/1.29.12/tools/shared.py#L958>`_).
 
-A few useful flags are:
-
-- 
-	.. _optimizing-code-no-exit-runtime:
-	
-	``NO_EXIT_RUNTIME``: Building with ``-s NO_EXIT_RUNTIME=1`` lets the compiler know that you don't want to shut down the runtime environment after the ``main()`` function finishes. This allows it to discard the ``atexit`` and global destructor calls it would otherwise make, improving code size and startup speed.
-
-	This is useful if your ``main()`` function finishes but you still want to execute code, for example in an app that uses a :ref:`main loop function <emscripten-runtime-environment-main-loop>`. 
-	
-	.. note:: Emscripten will not shut down the runtime if it detects :c:func:`emscripten_set_main_loop`, but it is better to optimise away the unnecessary code.
-
-
 
 Code size
 =========

site/source/docs/tools_reference/emcc.rst

@@ -405,7 +405,7 @@ Options that are modified or new in *emcc* are listed below:
 .. _emcc-emrun:
 
 ``--emrun``
-	Enables the generated output to be aware of the :ref:`emrun <Running-html-files-with-emrun>` command line tool. This allows ``stdout``, ``stderr`` and ``exit(returncode)`` capture when running the generated application through *emrun*.     
+	Enables the generated output to be aware of the :ref:`emrun <Running-html-files-with-emrun>` command line tool. This allows ``stdout``, ``stderr`` and ``exit(returncode)`` capture when running the generated application through *emrun*. (This enables `NO_EXIT_RUNTIME=0`, allowing normal runtime exiting with return code passing.)
 
 ``--cpuprofiler``
 	Embeds a simple CPU profiler onto the generated page. Use this to perform cursory interactive performance profiling.

src/library.js

@@ -596,6 +596,11 @@ LibraryManager.library = {
   atexit__proxy: 'sync',
   atexit__sig: 'ii',
   atexit: function(func, arg) {
+#if ASSERTIONS
+#if NO_EXIT_RUNTIME == 1
+    Runtime.warnOnce('atexit() called, but NO_EXIT_RUNTIME is set, so atexits() will not be called. set NO_EXIT_RUNTIME to 0 (see the FAQ)');
+#endif
+#endif
     __ATEXIT__.unshift({ func: func, arg: arg });
   },
   __cxa_atexit: 'atexit',

src/library_browser.js

@@ -1270,6 +1270,11 @@ var LibraryBrowser = {
   emscripten_force_exit__proxy: 'sync',
   emscripten_force_exit__sig: 'vi',
   emscripten_force_exit: function(status) {
+#if NO_EXIT_RUNTIME
+#if ASSERTIONS
+    Runtime.warnOnce('emscripten_force_exit cannot actually shut down the runtime, as the build has NO_EXIT_RUNTIME set');
+#endif
+#endif
     Module['noExitRuntime'] = false;
     Module['exit'](status);
   },

src/library_fs.js

@@ -1921,3 +1921,7 @@ mergeInto(LibraryManager.library, {
   }
 });
 
+if (FORCE_FILESYSTEM) {
+  DEFAULT_LIBRARY_FUNCS_TO_INCLUDE.push('$FS');
+}
+

src/library_syscall.js

@@ -779,7 +779,20 @@ var SyscallsLibrary = {
     return SYSCALLS.doReadv(stream, iov, iovcnt);
   },
 #if NO_FILESYSTEM
-  __syscall146__postset: '/* flush anything remaining in the buffer during shutdown */ __ATEXIT__.push(function() { var fflush = Module["_fflush"]; if (fflush) fflush(0); var printChar = ___syscall146.printChar; if (!printChar) return; var buffers = ___syscall146.buffers; if (buffers[1].length) printChar(1, {{{ charCode("\n") }}}); if (buffers[2].length) printChar(2, {{{ charCode("\n") }}}); });',
+  $flush_NO_FILESYSTEM: function() {
+    // flush anything remaining in the buffers during shutdown
+    var fflush = Module["_fflush"];
+    if (fflush) fflush(0);
+    var printChar = ___syscall146.printChar;
+    if (!printChar) return;
+    var buffers = ___syscall146.buffers;
+    if (buffers[1].length) printChar(1, {{{ charCode("\n") }}});
+    if (buffers[2].length) printChar(2, {{{ charCode("\n") }}});
+  },
+  __syscall146__deps: ['$flush_NO_FILESYSTEM'],
+#if NO_EXIT_RUNTIME == 0
+  __syscall146__postset: '__ATEXIT__.push(flush_NO_FILESYSTEM);',
+#endif
 #endif
   __syscall146: function(which, varargs) { // writev
 #if NO_FILESYSTEM == 0

src/postamble.js

@@ -212,8 +212,16 @@ Module['callMain'] = function callMain(args) {
     Module.realPrint('main() took ' + (Date.now() - start) + ' milliseconds');
 #endif
 
+#if EMTERPRETIFY_ASYNC
+    // if we are saving the stack, then do not call exit, we are not
+    // really exiting now, just unwinding the JS stack
+    if (EmterpreterAsync.state !== 1) {
+#endif // EMTERPRETIFY_ASYNC
     // if we're not running an evented main loop, it's time to exit
-    exit(ret, /* implicit = */ true);
+      exit(ret, /* implicit = */ true);
+#if EMTERPRETIFY_ASYNC
+    }
+#endif // EMTERPRETIFY_ASYNC
   }
   catch(e) {
     if (e instanceof ExitStatus) {
@@ -313,17 +321,62 @@ function run(args) {
 Module['run'] = run;
 
 function exit(status, implicit) {
-  if (implicit && Module['noExitRuntime']) {
 #if ASSERTIONS
-    Module.printErr('exit(' + status + ') implicitly called by end of main(), but noExitRuntime, so not exiting the runtime (you can use emscripten_force_exit, if you want to force a true shutdown)');
+#if NO_EXIT_RUNTIME == 1
+  // Compiler settings do not allow exiting the runtime, so flushing
+  // the streams is not possible. but in ASSERTIONS mode we check
+  // if there was something to flush, and if so tell the user they
+  // should request that the runtime be exitable.
+  // Normally we would not even include flush() at all, but in ASSERTIONS
+  // builds we do so just for this check, and here we see if there is any
+  // content to flush, that is, we check if there would have been
+  // something a non-ASSERTIONS build would have not seen.
+  // How we flush the streams depends on whether we are in NO_FILESYSTEM
+  // mode (which has its own special function for this; otherwise, all
+  // the code is inside libc)
+#if NO_FILESYSTEM
+  var flush = {{{ '$flush_NO_FILESYSTEM' in addedLibraryItems ? 'flush_NO_FILESYSTEM' : 'null' }}};
+#else
+  var flush = {{{ '$FS' in addedLibraryItems ? 'FS.quit' : "Module['_fflush']" }}};
 #endif
+  if (flush) {
+    var print = Module['print'];
+    var printErr = Module['printErr'];
+    var has = false;
+    Module['print'] = Module['printErr'] = function(x) {
+      has = true;
+    }
+    try { // it doesn't matter if it fails
+      flush(0);
+    } catch(e) {}
+    Module['print'] = print;
+    Module['printErr'] = printErr;
+    if (has) {
+      Runtime.warnOnce('stdio streams had content in them that was not flushed. you should set NO_EXIT_RUNTIME to 0 (see the FAQ), or make sure to emit a newline when you printf etc.');
+    }
+  }
+#endif // NO_EXIT_RUNTIME
+#endif // ASSERTIONS
+
+  // if this is just main exit-ing implicitly, and the status is 0, then we
+  // don't need to do anything here and can just leave. if the status is
+  // non-zero, though, then we need to report it.
+  // (we may have warned about this earlier, if a situation justifies doing so)
+  if (implicit && Module['noExitRuntime'] && status === 0) {
     return;
   }
 
   if (Module['noExitRuntime']) {
 #if ASSERTIONS
-    Module.printErr('exit(' + status + ') called, but noExitRuntime, so halting execution but not exiting the runtime or preventing further async execution (you can use emscripten_force_exit, if you want to force a true shutdown)');
-#endif
+    // if exit() was called, we may warn the user if the runtime isn't actually being shut down
+    if (!implicit) {
+#if NO_EXIT_RUNTIME
+      Module.printErr('exit(' + status + ') called, but NO_EXIT_RUNTIME is set, so halting execution but not exiting the runtime or preventing further async execution (build with NO_EXIT_RUNTIME=0, if you want a true shutdown)');
+#else
+      Module.printErr('exit(' + status + ') called, but noExitRuntime is set due to an async operation, so halting execution but not exiting the runtime or preventing further async execution (you can use emscripten_force_exit, if you want to force a true shutdown)');
+#endif // NO_EXIT_RUNTIME
+    }
+#endif // ASSERTIONS
   } else {
 #if USE_PTHREADS
     PThread.terminateAllThreads();

src/pthread-main.js

@@ -156,8 +156,7 @@ this.onmessage = function(e) {
       }
       // The thread might have finished without calling pthread_exit(). If so, then perform the exit operation ourselves.
       // (This is a no-op if explicit pthread_exit() had been called prior.)
-      if (!Module['noExitRuntime']) PThread.threadExit(result);
-      else console.log('pthread noExitRuntime: not quitting.');
+      PThread.threadExit(result);
     } else if (e.data.cmd === 'cancel') { // Main thread is asking for a pthread_cancel() on this thread.
       if (threadInfoStruct && PThread.thisThreadCancelState == 0/*PTHREAD_CANCEL_ENABLE*/) {
         PThread.threadCancel();

src/settings.js

@@ -44,8 +44,12 @@ var VERBOSE = 0; // When set to 1, will generate more verbose output during comp
 var INVOKE_RUN = 1; // Whether we will run the main() function. Disable if you embed the generated
                     // code in your own, and will call main() yourself at the right time (which you
                     // can do with Module.callMain(), with an optional parameter of commandline args).
-var NO_EXIT_RUNTIME = 0; // If set, the runtime is not quit when main() completes (allowing code to
-                         // run afterwards, for example from the browser main event loop).
+var NO_EXIT_RUNTIME = 1; // If 1, the runtime is not quit when main() completes (allowing code to
+                         // run afterwards, for example from the browser main event loop). atexit()s
+                         // are also not executed, and we can avoid including code for runtime shutdown,
+                         // like flushing the stdio streams.
+                         // Set this to 0 if you do want atexit()s or stdio streams to be flushed
+                         // on exit.
 var MEM_INIT_METHOD = 0; // How to represent the initial memory content.
                          // 0: embed a base64 string literal representing the initial memory data
                          // 1: create a *.mem file containing the binary data of the initial memory;