Installation issue with Tensorflow-cpu, no module named '_pwyrap_tensorflow_internal'

[Overdrivr]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 server in desktop mode
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): tensorflow-gpu 1.10.0
• Python version: Anaconda x64 3.6
• Bazel version (if compiling from source): NA
• GCC/Compiler version (if compiling from source): NA
• CUDA/cuDNN version: 9.0/7.0
• GPU model and memory: Q4000
• Exact command to reproduce: N/A

Describe the problem

1. Create a python application that imports tensorflow-gpu
2. Package the application with pyinstaller
3. Start the generated executable, that crashes while trying to import tensorflow.

The error trace is this:

Traceback (most recent call last):
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
  File "c:\users\paperspace\.virtualenvs\***-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in <module>
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in swig_import_helper
  File "importlib\__init__.py", line 126, in import_module
ModuleNotFoundError: No module named 'tensorflow.python._pywrap_tensorflow_internal'


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

On the other hand, doing the same process with tensorflow(-cpu) works fine.

I ruled out CUDA/cuDNN installation issue, because if I run my python application using python, it works fine with the GPU version.

I have tried checking at the .exe with dependency_walker, but appart from false-positive missing API-MS-* and EXT-MS-* DLLs, I don't see anything obvious.

So I'm stuck here.
What discrepancy could create an import issue with tensorflow-gpu and not with tensorflow-cpu ?

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Exact command to reproduce

[Overdrivr]

I've updated the form accordingly

[Overdrivr]

@ymodak I have made some progress. It turns out, I was building tensorflow from an Anaconda shell. I removed Anaconda and installed standard python 3.6.6 64 bits.

By doing so, the initial problem went away.
Executable runs fine on the build machine.
However, if I transfer the standalone application to another machine, it fails with the following error:

Traceback (most recent call last):
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in <module>
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in swig_import_helper
  File "importlib\__init__.py", line 126, in import_module
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "nxt\nxt.py", line 2, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\__init__.py", line 22, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\__init__.py", line 49, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in <module>
ImportError: Traceback (most recent call last):
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in <module>
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in swig_import_helper
  File "importlib\__init__.py", line 126, in import_module
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Here is the code to reproduce:

import tensorflow as tf

def main():	
	hello = tf.constant('hello TF')
	sess = tf.Session()
	print(sess.run(hello))

if __name__ == '__main__':
    main()

Steps to reproduce:

1. Run the program with python 3.6 and tensorflow-gpu 1.10.0. Works ok.
2. Package into a standalone with PyInstaller
3. Run the executable. Works ok.
4. Move the folder containing the executable and all dependencies to another machine.
5. Run the executable. Fails with error above.

[Overdrivr]

I finally managed to catch the exception happening at import time, application is raising EXCEPTION_ILLEGAL_INSTRUCTION.

Could this be related to AXV instructions not supported ? The processor of the VM is an Intel Xeon E5-2623 v4 that apparently supports AVX2.

[wt-huang]

@Overdrivr This is not a bug. You can run some sanity checks on the second machine to make sure the environment is set up correctly and dependencies are installed properly.

[Overdrivr]

You're correct, there is a configuration issue on the second machine.

I have installed python 3.6.6 on it, created a new virtual env, only containing tensorflow, and can reproduce the issue.

What puzzles me is that I have installed tensorflow-cpu on it, and I still get _pywrap_tensorflow_internal not found (And it's not the first time I've installed TF on a machine, I've done it for CPU and GPU version a good dozen of times).

I have ran the following sanity check (updated for tensorflow 1.10.0, Cuda 9.0, cuDNN 7.0.X):

import ctypes
import imp
import sys
from ctypes.util import find_library

def main():
    check()

def check():
    try:
        import tensorflow as tf
        print("TensorFlow successfully installed.")
        if tf.test.is_built_with_cuda():
            print("The installed version of TensorFlow includes GPU support.")
        else:
            print("The installed version of TensorFlow does not include GPU support.")
            #sys.exit(0)
    except ImportError as e:
        print("ERROR: Failed to import the TensorFlow module.")
        print('Reason: {}'.format(e))

    candidate_explanation = False

    python_version = sys.version_info.major, sys.version_info.minor
    print("- Python version is %d.%d." % python_version)
    if not (python_version == (3, 5) or python_version == (3, 6)):
        candidate_explanation = True
        print("- The official distribution of TensorFlow for Windows requires "
          "Python version 3.5 or 3.6.")

    try:
        _, pathname, _ = imp.find_module("tensorflow")
        print("- TensorFlow is installed at: %s" % pathname)
    except ImportError:
        candidate_explanation = False
        print("""- No module named TensorFlow is installed in this Python environment. You may
install it using the command `pip install tensorflow`.""")

    try:
        msvcp140 = ctypes.WinDLL("msvcp140.dll")
        msvcp140_path = find_library("msvcp140.dll")
        print('- msvcp140.dll Found at {}'.format(msvcp140_path))
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'msvcp140.dll'. You may install this DLL by downloading Microsoft Visual
  C++ 2015 Redistributable Update 3 from this URL:
  https://www.microsoft.com/en-us/download/details.aspx?id=53587""")

    try:
        cudart64_90 = ctypes.WinDLL("cudart64_90.dll")
        cudart64_90_path = find_library("cudart64_90.dll")
        print('- Cuda 9.0 found at {}'.format(cudart64_90_path))
        # TODO: Look for version.txt file in CUDA path
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'cudart64_90.dll'. Download and install CUDA 9.0 from
  this URL: https://developer.nvidia.com/cuda-toolkit""")

    try:
        nvcuda = ctypes.WinDLL("nvcuda.dll")
        nvcuda_path = find_library("nvcuda.dll")
        print('- nvcuda.dll found at {}'.format(nvcuda_path))
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'nvcuda.dll'. The GPU version of TensorFlow requires that
  this DLL be installed in a directory that is named in your %PATH%
  environment variable. Typically it is installed in 'C:\Windows\System32'.
  If it is not present, ensure that you have a CUDA-capable GPU with the
  correct driver installed.""")

    cudnn7_found = False
    try:
        cudnn7 = ctypes.WinDLL("cudnn64_7.dll")
        cudnn7_path = find_library("cudnn64_7.dll")
        print("- cuDNN Found at {}".format(cudnn7_path))
        cudnn7_found = True
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'cudnn64_7.dll'. The GPU version of TensorFlow
  requires that this DLL be installed in a directory that is named in
  your %PATH% environment variable. Note that installing cuDNN is a
  separate step from installing CUDA, and it is often found in a
  different directory from the CUDA DLLs. You may install the
  necessary DLL by downloading cuDNN 7.0 for Cuda 9.0 from this URL:
  https://developer.nvidia.com/cudnn""")

    if not candidate_explanation:
        print("""
- All required DLLs appear to be present. Please open an issue on the
  TensorFlow GitHub page: https://github.com/tensorflow/tensorflow/issues""")

    sys.exit(-1)


if __name__ == '__main__':
    main()

Output:

ERROR: Failed to import the TensorFlow module.
Reason: Traceback (most recent call last):
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 16, in swig_import_helper
    return importlib.import_module('_pywrap_tensorflow_internal')
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
- Python version is 3.6.
- TensorFlow is installed at: C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow
- msvcp140.dll Found at C:\Windows\system32\msvcp140.dll
- Cuda 9.0 found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudart64_90.dll
- nvcuda.dll found at C:\Windows\system32\nvcuda.dll
- cuDNN Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

- All required DLLs appear to be present. Please open an issue on the
  TensorFlow GitHub page: https://github.com/tensorflow/tensorflow/issues

Basically, the sanity check does not highlight any error, yet there seems to be an issue loading some DLL. This is very weird, because I have installed TF-cpu and GPU on another machine with exactly the same specs two hours before that one (They are both freshly created machines on a cloud provider with the same base Windows 10 image).

[Overdrivr]

I have tried TF 1.5 and it solves the issue. TF can be imported just fine. So, clearly, it has be an issue with AVX.

But it is stated here (and everywhere else) that the Xeon E5-2623 v4 supports AVX2, so I guess it should support AVX too, right ?

To be sure, I wrote a small script that fetches CPUID and checks if the hardware supports AVX.
Install cpuid to run it.

from cpuid import *

def _is_set(id, reg_idx, bit):
    regs = cpuid(id)

    if (1 << bit) & regs[reg_idx]:
        return "Yes"
    else:
        return "--"

print("Vendor ID         : %s" % cpu_vendor())
print("CPU name          : %s" % cpu_name())
print("Microarchitecture : %s%s" % cpu_microarchitecture())
print("Vector instructions supported:")
print("SSE       : %s" % _is_set(1, 3, 25))
print("SSE2      : %s" % _is_set(1, 3, 26))
print("SSE3      : %s" % _is_set(1, 2, 0))
print("SSSE3     : %s" % _is_set(1, 2, 9))
print("SSE4.1    : %s" % _is_set(1, 2, 19))
print("SSE4.2    : %s" % _is_set(1, 2, 20))
print("SSE4a     : %s" % _is_set(0x80000001, 2, 6))
print("AVX       : %s" % _is_set(1, 2, 28))
print("AVX2      : %s" % _is_set(7, 1, 5))
print("BMI1      : %s" % _is_set(7, 1, 3))
print("BMI2      : %s" % _is_set(7, 1, 8))

And this is what I get:

Vendor ID         : GenuineIntel
CPU name          : Intel(R) Xeon(R) CPU E5-2623 v4 @ 2.60GHz
Microarchitecture : broadwellnoavx
Vector instructions supported:
SSE       : Yes
SSE2      : Yes
SSE3      : Yes
SSSE3     : Yes
SSE4.1    : Yes
SSE4.2    : Yes
SSE4a     : --
AVX       : --
AVX2      : --
BMI1      : Yes
BMI2      : Yes

Just the name of the micro-architecture is pretty self-explanatory.
Not only does he not support AVX2, he does not even support AVX1.

I would like to give a big THANKS to intel for advertising their processor to support AVX when they clearly don't (and for wasting our time).

[Overdrivr]

While we're at it, I'm getting the same problem with tensorflow-cpu... on another freshly created machine.. that supports AVX. Fun ride.

I'm leaving this here for documentation, it might help others.

In my production virtualenv with tensorflow-cpu + a bunch of other packages, I get the following error (the output comes from a tensorflow-install-checker tool that I will release in public domain):

INFO - Vendor ID         : GenuineIntel
INFO - CPU name          : Intel(R) Xeon(R) CPU E5-2623 v4 @ 2.60GHz
INFO - Microarchitecture : broadwell
OK  - Your processor supports AVX instructions
ERR - Failed to import the TensorFlow module.
Traceback (most recent call last):
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 16, in swig_import_helper
    return importlib.import_module('_pywrap_tensorflow_internal')
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
OK  - TensorFlow found at: c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow
OK  - msvcp140.dll found at: c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\scipy\extra-dll\msvcp140.dll
OK  - Cuda 9.0 found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudart64_90.dll
OK  - nvcuda.dll found at C:\Windows\system32\nvcuda.dll
OK  - cuDNN Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

In another virtualenv with only tensorflow-cpu, I get this one:

Traceback (most recent call last):
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\platform\self_check.py", line 47, in preload_check
    ctypes.WinDLL(build_info.msvcp_dll_name)
  File "C:\Program Files\Python36\Lib\ctypes\__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\__init__.py", line 22, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 30, in <module>
    self_check.preload_check()
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\platform\self_check.py", line 55, in preload_check
    % build_info.msvcp_dll_name)
ImportError: Could not find 'msvcp140.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. You may install this DLL by downloading Visual C++ 2015 Redistributable Update 3 from this URL: https://www.microsoft.com/en-us/download/details.aspx?id=53587

Hum weird right ? Let's summarize:

• In the first env, msvcp140.dll is found, but TF fails to load
• In the second env, msvcp140.dll is missing

If you look closely, the msvcp140.dll found in the first env does not come from system32 folder.
It comes from scipy (c:\users\paperspace\.virtualenvs\******-jrq2aiwp\lib\site-packages\scipy\extra-dll\msvcp140.dll).
Scipy is (to solve other issues) releasing msvcp140.dll in their extra-dll folder. This folder ends up in the PATH, and the dll in question picked up by the library finder.

Now, why that's a problem:

• For machines that already have this DLL in system32 folder, it's fine because system path takes precedence over user path.
• For machines without this DLL, instead of having TF crash with DLL missing, it crashes with some cryptic _pywrap_tensorflow_internal... error, most likely because the DLL found in scipy is different and not compatible that the latest installed by Microsoft Visual C++ 2015 Redistributable Update 3 RC. The workaround is then to simply install Microsoft Visual C++ 2015 Redistributable 64 bits. This way, the DLL is installed in system32 folder and takes precedences over the one installed by scipy.

After doing so on my VM, importing TF in both virtualenvs now runs properly. 🥂

INFO - Vendor ID         : GenuineIntel
INFO - CPU name          : Intel(R) Xeon(R) CPU E5-2623 v4 @ 2.60GHz
INFO - Microarchitecture : broadwell
OK  - Your processor supports AVX instructions
OK  - TensorFlow is functional.
ERR - TensorFlow does not include GPU support.
OK  - TensorFlow found at: c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow
OK  - msvcp140.dll found at: C:\Windows\system32\msvcp140.dll
OK  - Cuda 9.0 found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudart64_90.dll
OK  - nvcuda.dll found at C:\Windows\system32\nvcuda.dll
OK  - cuDNN Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Maybe it might be worth mentioning it in the install docs.