'ue4 conan generate' fails on 5.1.0

[jlsalmon]

(Moved from adamrehn/ue4-docker#281)

Output of the ue4-docker info command:

ue4-docker version:         0.0.102 (latest available version is 0.0.102)
Operating system:           Linux (Ubuntu 20.04.3 LTS, 5.11.0-1028-aws)
Docker daemon version:      20.10.12
NVIDIA Docker supported:    No
Maximum image size:         No limit detected
Available disk space:       491.4 GiB
Total system memory:        30.35 GiB physical, 6.39 GiB virtual
CPU:                        8 physical, 16 logical (x86_64)

Building 5.1.0 fails at the ue4 conan generate step:

Copying the "ue5.1-Linux-x86_64-unknown-linux-gnu" Conan profile into a new profile named "ue4"...
Retrieving thirdparty library list from UBT...
Traceback (most recent call last):
  File "/usr/local/bin/ue4", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/ue4cli/cli.py", line 222, in main
    SUPPORTED_COMMANDS[command]['action'](manager, args)
  File "/usr/local/lib/python3.8/dist-packages/conan_ue4cli/main.py", line 48, in main
    SUBCOMMANDS[subcommand]['function'](manager, args[1:])
  File "/usr/local/lib/python3.8/dist-packages/conan_ue4cli/commands/generate.py", line 153, in generate
    libs = [lib for lib in manager.listThirdPartyLibs() if lib != 'libc++']
  File "/usr/local/lib/python3.8/dist-packages/ue4cli/UnrealManagerBase.py", line 183, in listThirdPartyLibs
    return interrogator.list(self.getPlatformIdentifier(), configuration, self._getLibraryOverrides())
  File "/usr/local/lib/python3.8/dist-packages/ue4cli/UE4BuildInterrogator.py", line 20, in list
    modules = self._getThirdPartyLibs(platformIdentifier, configuration)
  File "/usr/local/lib/python3.8/dist-packages/ue4cli/UE4BuildInterrogator.py", line 170, in _getThirdPartyLibs
    self.runUBTFunc('UnrealEditor', platformIdentifier, configuration, args)
  File "/usr/local/lib/python3.8/dist-packages/ue4cli/UnrealManagerBase.py", line 695, in <lambda>
    ubtLambda = lambda target, platform, config, args: self._runUnrealBuildTool(target, platform, config, args, True)
  File "/usr/local/lib/python3.8/dist-packages/ue4cli/UnrealManagerBase.py", line 687, in _runUnrealBuildTool
    return Utility.capture(arguments, cwd=self.getEngineRoot(), raiseOnError=True)
  File "/usr/local/lib/python3.8/dist-packages/ue4cli/Utility.py", line 125, in capture
    raise Exception(
Exception: child process ['/home/ue4/UnrealEngine/Engine/Build/BatchFiles/Linux/Build.sh', 'UnrealEditor', 'Linux', 'Development', '-Mode=JsonExport', '-OutputFile=/tmp/tmps0jdktyq/ubt_output.json'] failed with exit code 1
stdout: "Setting up bundled DotNet SDK
Running command : dotnet Engine/Binaries/DotNET/UnrealBuildTool/UnrealBuildTool.dll UnrealEditor Linux Development -Mode=JsonExport -OutputFile=/tmp/tmps0jdktyq/ubt_output.json
Welcome to .NET !
---------------------
SDK Version:
Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.
Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
"
stderr: "Could not execute because the specified command or file was not found.
Possible reasons for this include:
  * You misspelled a built-in dotnet command.
  * You intended to execute a .NET program, but dotnet-Engine/Binaries/DotNET/UnrealBuildTool/UnrealBuildTool.dll does not exist.
  * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
"
The command '/bin/sh -c ue4 conan generate' returned a non-zero code: 1

Could this have something to do with UBT needing to be pre-built?

[russkel]

I am also running into this. I had a look and mine is using the source image which does not contain a precompiled DLL with that name. Changing to minimal doesn't work because generate cannot use an installed build of UE apparently.

[russkel]

I hacked in a copy from the minimal image of the UnrealBuildTool directory and now the next error is:

#14 [conan 6/6] RUN ue4 conan generate && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that 
for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo ''
#14 sha256:9586126ecafb49460afac2b01fb0b22e089cbbf4b1a05709303b7dda46438cb5
#14 0.457 Using user-specified engine root: /home/ue4/UnrealEngine                                                                                          
#14 6.792 Traceback (most recent call last):
#14 6.792   File "/usr/local/bin/ue4", line 8, in <module>
#14 6.792     sys.exit(main())                                                                                                                              
#14 6.792   File "/usr/local/lib/python3.10/dist-packages/ue4cli/cli.py", line 222, in main
#14 6.792 Removing the "ue5.1-Linux-x86_64-unknown-linux-gnu" Conan profile if it already exists...                                                         #14 6.792 Removing any previous versions of profile base packages...
#14 6.792 Creating "ue5.1-Linux-x86_64-unknown-linux-gnu" Conan profile using autodetected settings...
#14 6.792 Installing profile base packages...                           
#14 6.792 Copying the "ue5.1-Linux-x86_64-unknown-linux-gnu" Conan profile into a new profile named "ue4"...
#14 6.792 Retrieving thirdparty library list from UBT...                                                                                                    
#14 6.792 Removing any previous versions of generated wrapper packages for 5.1...                                                                           #14 6.792 Generating and installing toolchain wrapper package...
#14 6.792   Wrapping clang: /home/ue4/UnrealEngine/Engine/Extras/ThirdPartyNotUE/SDKs/HostLinux/Linux_x64/v20_clang-13.0.1-centos7/x86_64-unknown-linux-gnu/
bin/clang                                                                     
#14 6.792   Wrapping lib++: /home/ue4/UnrealEngine/Engine/Source/ThirdParty/Unix/LibCxx/lib/Unix/x86_64-unknown-linux-gnu/libc++.a
#14 6.792     SUPPORTED_COMMANDS[command]['action'](manager, args)                                                                                          #14 6.792   File "/usr/local/lib/python3.10/dist-packages/conan_ue4cli/main.py", line 48, in main
#14 6.792     SUBCOMMANDS[subcommand]['function'](manager, args[1:])
#14 6.792   File "/usr/local/lib/python3.10/dist-packages/conan_ue4cli/commands/generate.py", line 169, in generate
#14 6.792     PackageManagement.install(join(packagesDir, 'toolchain-wrapper'), channel, profile, [
#14 6.792   File "/usr/local/lib/python3.10/dist-packages/conan_ue4cli/common/PackageManagement.py", line 16, in install
#14 6.792     return Utility.run(['conan', 'create', '.', 'adamrehn/' + channel, '--profile=' + profile] + args, cwd=packageDir)
#14 6.792   File "/usr/local/lib/python3.10/dist-packages/conan_ue4cli/common/Utility.py", line 29, in run
#14 6.792     raise Exception(                                                                                                                              
#14 6.792 Exception: child process ['conan', 'create', '.', 'adamrehn/5.1', '--profile=ue5.1-Linux-x86_64-unknown-linux-gnu', '--env', 'WRAPPED_TOOLCHAIN=/home/ue4/UnrealEngine/Engine/Extras/ThirdPartyNotUE/SDKs/HostLinux/Linux_x64/v20_clang-13.0.1-centos7/x86_64-unknown-linux-gnu', '--env', 'WRAPPED_LIBCXX=/ho
me/ue4/UnrealEngine/Engine/Source/ThirdParty/Unix/LibCxx'] failed with exit code 1
#14 6.792 stdout: "Exporting package recipe                                                                                                                 #14 6.792 toolchain-wrapper/ue4@adamrehn/5.1 exports: Copied 4 '.py' files: conanfile.py, clang.py, clang++.py, ClangInterposition.py                       #14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: A new conanfile.py version was exported
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Folder: /home/ue4/.conan/data/toolchain-wrapper/ue4/adamrehn/5.1/export
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Exported revision: 3845df134b0b0b303c03d66020e6c06c                                                           #14 6.792 Configuration:                                                                                                                                    #14 6.792 [settings]
#14 6.792 arch=x86_64                                                         
#14 6.792 arch_build=x86_64
#14 6.792 build_type=Release
#14 6.792 compiler=clang                                                                                                                                    #14 6.792 compiler.libcxx=libc++                                                                                                                            
#14 6.792 compiler.version=13
#14 6.792 os=Linux                                                                                                                                          #14 6.792 os_build=Linux                                                                                                                                    
#14 6.792 [options]                                                                                                                                         
#14 6.792 [build_requires]
#14 6.792 *: toolchain-wrapper/ue4@adamrehn/5.1                                                                                                             
#14 6.792 [env]
#14 6.792 UNREAL_ENGINE_VERSION=5.1
#14 6.792 WRAPPED_LIBCXX=/home/ue4/UnrealEngine/Engine/Source/ThirdParty/Unix/LibCxx
#14 6.792 WRAPPED_TOOLCHAIN=/home/ue4/UnrealEngine/Engine/Extras/ThirdPartyNotUE/SDKs/HostLinux/Linux_x64/v20_clang-13.0.1-centos7/x86_64-unknown-linux-gnu
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Forced build from source
#14 6.792 Installing package: toolchain-wrapper/ue4@adamrehn/5.1
#14 6.792 Requirements
#14 6.792     toolchain-wrapper/ue4@adamrehn/5.1 from local cache - Cache
#14 6.792 Packages
#14 6.792     toolchain-wrapper/ue4@adamrehn/5.1:7398dab5728f06866bd82627d2e411754f86297a - Build
#14 6.792 
#14 6.792 Installing (downloading, building) binaries...
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Configuring sources in /home/ue4/.conan/data/toolchain-wrapper/ue4/adamrehn/5.1/source
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Copying sources to build folder
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Building your package in /home/ue4/.conan/data/toolchain-wrapper/ue4/adamrehn/5.1/build/7398dab5728f06866bd826
27d2e411754f86297a
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Generator txt created conanbuildinfo.txt
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Aggregating env generators
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Calling build()
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Package '7398dab5728f06866bd82627d2e411754f86297a' built
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Build folder /home/ue4/.conan/data/toolchain-wrapper/ue4/adamrehn/5.1/build/7398dab5728f06866bd82627d2e411754f
86297a
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Generated conaninfo.txt
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Generated conanbuildinfo.txt
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Generating the package
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Package folder /home/ue4/.conan/data/toolchain-wrapper/ue4/adamrehn/5.1/package/7398dab5728f06866bd82627d2e411
754f86297a
#14 6.792 toolchain-wrapper/ue4@adamrehn/5.1: Calling package()
#14 6.792 "
#14 6.792 stderr: "toolchain-wrapper/ue4@adamrehn/5.1: WARN: This conanfile has no build step
#14 6.792 ERROR: toolchain-wrapper/ue4@adamrehn/5.1: Error in package() method, line 69
#14 6.792       libraries = dirname(self._find_libcxx(libcxx, architecture))
#14 6.792 while calling '_find_libcxx', line 42
#14 6.792       raise RuntimeError('Failed to locate libc++.a for architecture "{}" inside directory "{}"!'.format(architecture, root))
#14 6.792       RuntimeError: Failed to locate libc++.a for architecture "x86_64" inside directory "/home/ue4/UnrealEngine/Engine/Source/ThirdParty/Unix/Lib
Cxx"!

Quick check finds this:

ue4@895cb72076f3:~/UnrealEngine/Engine/Source/ThirdParty$ find . | grep libc++.a
./Unix/LibCxx/lib/Unix/aarch64-unknown-linux-gnueabi/libc++.a
./Unix/LibCxx/lib/Unix/x86_64-unknown-linux-gnu/libc++.a

While looking in the Unreal 4.27.2 image I found this:

ue4@abee626cafda:~/UnrealEngine/Engine/Source/ThirdParty$ find . | grep libc++.a
./Linux/LibCxx/lib/Linux/aarch64-unknown-linux-gnueabi/libc++.a
./Linux/LibCxx/lib/Linux/x86_64-unknown-linux-gnu/libc++.a

Looks like the file names have changed in this toolchain version, sigh.

[petergerten]

What would be the best way to incorporate the fix in #21 ?
Unfortunately it is not in a pip release yet, so just using ue4-docker installs an old version without that fix.
This currently prevents me from building UE 5.0.3 or 5.1.

[jlsalmon]

I haven't tested that fix, but it would be straightforward to monkey patch conan-ue4cli in the absence of this being released.

[petergerten]

For a local conan-ue4cli I agree, but I am actually only using ue4-docker where it builds some Dockerfile using it. Not sure how the inner workings of ue4-docker are.

[russkel]

I think I modified the Dockerfile part where it pip installed it to pip install https://github.com/Greenroom-Robotics/conan-ue4cli/archive/refs/heads/feature/fix-libcxx.zip

According to this https://github.com/adamrehn/ue4-docker/blob/c4e8ecec9b0f1dfdee0d6fa96cd4592c6a016644/ue4docker/infrastructure/BuildConfiguration.py#L225

You could pass an argument to ue4-docker: -conan-ue4cli=https://github.com/Greenroom-Robotics/conan-ue4cli/archive/refs/heads/feature/fix-libcxx.zip.

[petergerten]

You could pass an argument to ue4-docker: -conan-ue4cli=https://github.com/Greenroom-Robotics/conan-ue4cli/archive/refs/heads/feature/fix-libcxx.zip.

I tried that (earlier I tried these two patches as well) - but still get this error (both on 5.0.3 and 5.1.0):

----
 > [conan 5/5] RUN ue4 conan generate && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo '':                                                              
#13 4.305 Removing the "ue5.1-Linux-x86_64-unknown-linux-gnu" Conan profile if it already exists...             
#13 4.305 Removing any previous versions of profile base packages...                                            
#13 4.305 Creating "ue5.1-Linux-x86_64-unknown-linux-gnu" Conan profile using autodetected settings...          
#13 4.305 Installing profile base packages...                                                                   
#13 4.305 Copying the "ue5.1-Linux-x86_64-unknown-linux-gnu" Conan profile into a new profile named "ue4"...    
#13 4.305 Retrieving thirdparty library list from UBT...                                                        
#13 4.305 Using user-specified engine root: /home/ue4/UnrealEngine
#13 4.306 Traceback (most recent call last):
#13 4.306   File "/usr/local/bin/ue4", line 8, in <module>
#13 4.306     sys.exit(main())
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/ue4cli/cli.py", line 222, in main
#13 4.306     SUPPORTED_COMMANDS[command]['action'](manager, args)
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/conan_ue4cli/main.py", line 48, in main
#13 4.306     SUBCOMMANDS[subcommand]['function'](manager, args[1:])
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/conan_ue4cli/commands/generate.py", line 153, in generate
#13 4.306     libs = [lib for lib in manager.listThirdPartyLibs() if lib != 'libc++']
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/ue4cli/UnrealManagerBase.py", line 183, in listThirdPartyLibs
#13 4.306     return interrogator.list(self.getPlatformIdentifier(), configuration, self._getLibraryOverrides())
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/ue4cli/UE4BuildInterrogator.py", line 20, in list
#13 4.306     modules = self._getThirdPartyLibs(platformIdentifier, configuration)
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/ue4cli/UE4BuildInterrogator.py", line 170, in _getThirdPartyLibs
#13 4.306     self.runUBTFunc('UnrealEditor', platformIdentifier, configuration, args)
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/ue4cli/UnrealManagerBase.py", line 695, in <lambda>
#13 4.306     ubtLambda = lambda target, platform, config, args: self._runUnrealBuildTool(target, platform, config, args, True)
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/ue4cli/UnrealManagerBase.py", line 687, in _runUnrealBuildTool
#13 4.306     return Utility.capture(arguments, cwd=self.getEngineRoot(), raiseOnError=True)
#13 4.306   File "/usr/local/lib/python3.8/dist-packages/ue4cli/Utility.py", line 125, in capture
#13 4.306     raise Exception(
#13 4.306 Exception: child process ['/home/ue4/UnrealEngine/Engine/Build/BatchFiles/Linux/Build.sh', 'UnrealEditor', 'Linux', 'Development', '-Mode=JsonExport', '-OutputFile=/tmp/tmpsisfjm1z/ubt_output.json'] failed with exit code 1
#13 4.306 stdout: "Setting up bundled DotNet SDK
#13 4.306 Running command : dotnet Engine/Binaries/DotNET/UnrealBuildTool/UnrealBuildTool.dll UnrealEditor Linux Development -Mode=JsonExport -OutputFile=/tmp/tmpsisfjm1z/ubt_output.json
#13 4.306 
#13 4.306 Welcome to .NET !
#13 4.306 ---------------------
#13 4.306 SDK Version: 
#13 4.306 
#13 4.306 Telemetry
#13 4.306 ---------
#13 4.306 The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.
#13 4.306 
#13 4.306 Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
#13 4.306 
#13 4.306 ----------------
#13 4.306 Installed an ASP.NET Core HTTPS development certificate.
#13 4.306 To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
#13 4.306 Learn about HTTPS: https://aka.ms/dotnet-https
#13 4.306 ----------------
#13 4.306 Write your first app: https://aka.ms/dotnet-hello-world
#13 4.306 Find out what's new: https://aka.ms/dotnet-whats-new
#13 4.306 Explore documentation: https://aka.ms/dotnet-docs
#13 4.306 Report issues and find source on GitHub: https://github.com/dotnet/core
#13 4.306 Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
#13 4.306 --------------------------------------------------------------------------------------
#13 4.306 "
#13 4.306 stderr: "Could not execute because the specified command or file was not found.
#13 4.306 Possible reasons for this include:
#13 4.306   * You misspelled a built-in dotnet command.
#13 4.306   * You intended to execute a .NET program, but dotnet-Engine/Binaries/DotNET/UnrealBuildTool/UnrealBuildTool.dll does not exist.
#13 4.306   * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
#13 4.306 "
----

[russkel]

I came across this as well. Engine/Binaries/DotNET/UnrealBuildTool/UnrealBuildTool.dll is not compiled in that image. It's all a bit of a mess. I had to copy the compiled UnrealBuildTool from the minimal image.

You can see what I did here in this commit Greenroom-Robotics/ue4-docker@04a48e4

You will have to change it to reflect your image names.

[petergerten]

thanks, that worked :-)