Compile with hidden visibility always; set via cmake property rather than compiler flag

This updates the compilation to always apply hidden visibility to
resolve the issues with default visibility causing problems under debug
compilations.  Moreover using the cmake property makes it easier for a
caller to override if absolutely needed for some reason.

For `pybind11_add_module` we use cmake to set the property; for the
targets, we append to compilation option to non-MSVC compilers.

Author: jagerman

CMakeLists.txt

@@ -91,6 +91,9 @@ if(NOT (CMAKE_VERSION VERSION_LESS 3.0))  # CMake >= 3.0
                                                 $<BUILD_INTERFACE:${PYTHON_INCLUDE_DIRS}>
                                                 $<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>)
   target_compile_options(pybind11 INTERFACE $<BUILD_INTERFACE:${PYBIND11_CPP_STANDARD}>)
+  if(NOT MSVC)
+    target_compile_options(pybind11 INTERFACE -fvisibility=hidden)
+  endif()
 
   add_library(module INTERFACE)
   add_library(pybind11::module ALIAS module)

docs/advanced/misc.rst

@@ -135,22 +135,16 @@ has been executed:
 
 Naturally, both methods will fail when there are cyclic dependencies.
 
-Note that compiling code which has its default symbol visibility set to
-*hidden* (e.g. via the command line flag ``-fvisibility=hidden`` on GCC/Clang) can interfere with the
-ability to access types defined in another extension module. Workarounds
-include changing the global symbol visibility (not recommended, because it will
-lead unnecessarily large binaries) or manually exporting types that are
-accessed by multiple extension modules:
+Note that pybind11 code compiled with hidden-by-default symbol visibility (e.g.
+via the command line flag ``-fvisibility=hidden`` on GCC/Clang), which is
+required proper pybind11 functionality, can interfere with the ability to
+access types defined in another extension module.  Working around this requires
+manually exporting types that are accessed by multiple extension modules;
+pybind11 provides a macro to do just this:
 
 .. code-block:: cpp
 
-    #ifdef _WIN32
-    #  define EXPORT_TYPE __declspec(dllexport)
-    #else
-    #  define EXPORT_TYPE __attribute__ ((visibility("default")))
-    #endif
-
-    class EXPORT_TYPE Dog : public Animal {
+    class PYBIND11_EXPORT Dog : public Animal {
         ...
     };
 

docs/compiling.rst

@@ -74,13 +74,15 @@ removes this target from the default build (see CMake docs for details).
 
 Since pybind11 is a template library, ``pybind11_add_module`` adds compiler
 flags to ensure high quality code generation without bloat arising from long
-symbol names and duplication of code in different translation units. The
-additional flags enable LTO (Link Time Optimization), set default visibility
-to *hidden* and strip unneeded symbols. See the :ref:`FAQ entry <faq:symhidden>`
-for a more detailed explanation. These optimizations are never applied in
-``Debug`` mode. If ``NO_EXTRAS`` is given, they will always be disabled, even
-in ``Release`` mode. However, this will result in code bloat and is generally
-not recommended.
+symbol names and duplication of code in different translation units. It
+sets default visibility to *hidden*, which is required for some pybind11
+features and functionality when attempting to load multiple pybind11 modules
+compiled under different pybind11 versions.  It also adds additional flags
+enabling LTO (Link Time Optimization) and strip unneeded symbols. See the
+:ref:`FAQ entry <faq:symhidden>` for a more detailed explanation. These
+latter optimizations are never applied in ``Debug`` mode.  If ``NO_EXTRAS`` is
+given, they will always be disabled, even in ``Release`` mode. However, this
+will result in code bloat and is generally not recommended.
 
 As stated above, LTO is enabled by default. Some newer compilers also support
 different flavors of LTO such as `ThinLTO`_. Setting ``THIN_LTO`` will cause
@@ -181,9 +183,8 @@ to an independently constructed (through ``add_library``, not
     flags (i.e. this is up to you).
 
     These include Link Time Optimization (``-flto`` on GCC/Clang/ICPC, ``/GL``
-    and ``/LTCG`` on Visual Studio). Default-hidden symbols on GCC/Clang/ICPC
-    (``-fvisibility=hidden``) and .OBJ files with many sections on Visual Studio
-    (``/bigobj``). The :ref:`FAQ <faq:symhidden>` contains an
+    and ``/LTCG`` on Visual Studio) and .OBJ files with many sections on Visual
+    Studio (``/bigobj``).  The :ref:`FAQ <faq:symhidden>` contains an
     explanation on why these are needed.
 
 Embedding the Python interpreter

docs/faq.rst

@@ -151,6 +151,33 @@ specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
 culprit is generally the generation of function signatures at compile time
 using C++14 template metaprogramming.
 
+.. _`faq:hidden_visibility`:
+
+"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]"
+============================================================================================================
+
+This error typically indicates that you are compiling without the required
+``-fvisibility`` flag.  pybind11 code internally forces hidden visibility on
+all internal code, but if non-hidden (and thus *exported*) code attempts to
+include a pybind type (for example, ``py::object`` or ``py::list``) you can run
+into this warning.
+
+To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
+compiling pybind code.
+
+As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
+have been compiled under different versions of pybind itself, it is also
+important that the symbols defined in one module do not clash with the
+potentially-incompatible symbols defined in another.  While Python extension
+modules are usually loaded with localized symbols (under POSIX systems
+typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
+can be changed, but even if it isn't it is not always enough to guarantee
+complete independence of the symbols involved when not using
+``-fvisibility=hidden``.
+
+Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size
+savings.  (See the following section for more details).
+
 
 .. _`faq:symhidden`:
 
@@ -192,11 +219,14 @@ world. So we'll generally only want to export symbols for those functions which
 are actually called from the outside.
 
 This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
-and Clang, which sets the default symbol visibility to *hidden*. It's best to
-do this only for release builds, since the symbol names can be helpful in
-debugging sessions. On Visual Studio, symbols are already hidden by default, so
-nothing needs to be done there. Needless to say, this has a tremendous impact
-on the final binary size of the resulting extension library.
+and Clang, which sets the default symbol visibility to *hidden*, which has a
+tremendous impact on the final binary size of the resulting extension library.
+(On Visual Studio, symbols are already hidden by default, so nothing needs to
+be done there.)
+
+In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
+potential serious issues when loading multiple modules and is required for
+proper pybind operation.  See the previous FAQ entry for more details.
 
 Another aspect that can require a fair bit of code are function signature
 descriptions. pybind11 automatically generates human-readable function

tools/pybind11Tools.cmake

@@ -139,6 +139,13 @@ function(pybind11_add_module target_name)
   set_target_properties(${target_name} PROPERTIES PREFIX "${PYTHON_MODULE_PREFIX}")
   set_target_properties(${target_name} PROPERTIES SUFFIX "${PYTHON_MODULE_EXTENSION}")
 
+  # -fvisibility=hidden is required to allow multiple modules compiled against
+  # different pybind versions to work properly, and for some features (e.g.
+  # py::module_local).  We force it on everything inside the `pybind11`
+  # namespace; also turning it on for a pybind module compilation here avoids
+  # potential warnings or issues from having mixed hidden/non-hidden types.
+  set_target_properties(${target_name} PROPERTIES CXX_VISIBILITY_PRESET "hidden")
+
   if(WIN32 OR CYGWIN)
     # Link against the Python shared library on Windows
     target_link_libraries(${target_name} PRIVATE ${PYTHON_LIBRARIES})
@@ -175,9 +182,6 @@ function(pybind11_add_module target_name)
   _pybind11_add_lto_flags(${target_name} ${ARG_THIN_LTO})
 
   if (NOT MSVC AND NOT ${CMAKE_BUILD_TYPE} MATCHES Debug)
-    # Set the default symbol visibility to hidden (very important to obtain small binaries)
-    target_compile_options(${target_name} PRIVATE "-fvisibility=hidden")
-
     # Strip unnecessary sections of the binary on Linux/Mac OS
     if(CMAKE_STRIP)
       if(APPLE)