Development environment fails to build basicsr

[dnoliver]

Describe the bug

Trying to deploy a development environment in Meteor Lake.
Run into a problem was while running the following command: https://github.com/intel/AI-Playground/blob/main/readme.md?plain=1#L57

cd service
pip install -r .\requirements-ultra.txt

The root cause (I think) is that basicsr is using a legacy format of setup.py.
To solve that, I had to install dependencies following the recommended command in the log: pip install --use-pep517
I think updating the docs might be needed.

To Reproduce

Steps to reproduce the behavior:

1. Follow the instructions in the readme file to create the development environment.

Expected behavior

Instructions on readme.md for development environment setup should work

Screenshots

Sharing logs:

(aipg_xpu) PS C:\Users\Nicolas Oliver\Downloads\AI-Playground\service> python -m pip install -r .\requirements-ultra.txt
Looking in indexes: https://pypi.org/simple, https://pytorch-extension.intel.com/release-whl/stable/mtl/us/
Collecting basicsr==1.4.2 (from -r .\requirements-ultra.txt (line 1))
  Using cached basicsr-1.4.2.tar.gz (172 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [44 lines of output]
      C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!

              ********************************************************************************
              Requirements should be satisfied by a PEP 517 installer.
              If you are using pip, you can try `pip install --use-pep517`.
              ********************************************************************************

      !!
        dist.fetch_build_eggs(dist.setup_requires)
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Nicolas Oliver\AppData\Local\Temp\pip-install-q11r13x8\basicsr_8b78bceb4d094ae98f2d8813e165273c\setup.py", line 147, in <module>
          setup(
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\__init__.py", line 116, in setup
          _install_setup_requires(attrs)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\__init__.py", line 89, in _install_setup_requires
          _fetch_build_eggs(dist)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\__init__.py", line 94, in _fetch_build_eggs
          dist.fetch_build_eggs(dist.setup_requires)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\dist.py", line 617, in fetch_build_eggs
          return _fetch_build_eggs(self, requires)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\installer.py", line 39, in _fetch_build_eggs
          resolved_dists = pkg_resources.working_set.resolve(
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\pkg_resources\__init__.py", line 897, in resolve
          dist = self._resolve_dist(
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\pkg_resources\__init__.py", line 933, in _resolve_dist
          dist = best[req.key] = env.best_match(
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\pkg_resources\__init__.py", line 1271, in best_match
          return self.obtain(req, installer)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\pkg_resources\__init__.py", line 1307, in obtain
          return installer(requirement) if installer else None
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\installer.py", line 107, in _fetch_build_egg_no_warn
          wheel.install_as_egg(dist_location)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\wheel.py", line 122, in install_as_egg
          self._install_as_egg(destination_eggdir, zf)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\wheel.py", line 130, in _install_as_egg
          self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\wheel.py", line 150, in _convert_metadata
          _unpack_zipfile_obj(zf, destination_eggdir)
        File "C:\Users\Nicolas Oliver\.conda\envs\aipg_xpu\lib\site-packages\setuptools\archive_util.py", line 131, in _unpack_zipfile_obj
          with open(target, 'wb') as f:
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Nicolas Oliver\\AppData\\Local\\Temp\\pip-install-q11r13x8\\basicsr_8b78bceb4d094ae98f2d8813e165273c\\.eggs\\torch-2.4.1-py3.10-win-amd64.egg\\torch\\include\\ATen\\ops\\_fake_quantize_per_tensor_affine_cachemask_tensor_qparams_compositeexplicitautograd_dispatch.h'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details

Environment (please complete the following information):

• OS: Windows11
• GPU: [e.g. Intel Arc A770 16G]
• CPU: Intel Core Ultra 7 155H
• Version: tip of the main branch, commit d23a807

Additional context

There are additional problems with the basicsr library, this is the first one I encountered

[Nuullll]

Could you please check if you have enough disk space when executing pip install?

[dnoliver]

Disk space looks fine:

(aipg_xpu) PS C:\Users\Nicolas Oliver\Downloads\AI-Playground\service> Get-PSDrive C  

Name           Used (GB)     Free (GB) Provider      Root                                                                                                                CurrentLocation 
----           ---------     --------- --------      ----                                                                                                                --------------- 
C                 288.61        187.46 FileSystem    C:\                                                                            Users\Nicolas Oliver\Downloads\AI-Playground\service

So today I tried to reproduce it, but I couldn't do it until I completely remove all the conda and pip cache, and force the basicsr lib to rebuild again. I am guessing that once you have a successful build, pip starts using it, so you don't go to its setup.py, and you don't run into the issue.

So to reproduce, you need a clean state, I did:

conda remove -n aipg_xpu --all
conda clean --all
pip cache purge

I ended up removing the %LOCALAPPDATA%\pip\cache folder entirely as well, because I was still getting a cached version of the built wheel. But yeah, after all that cleanup, it is reproducible 100%.

[Nuullll]

Okay, found another similar issue XPixelGroup/BasicSR#571, then it's possibly due to file path length limitation on windows:

ChatGPT: File Path Length on Windows: The error FileNotFoundError: No such file or directory could be caused by an excessively long file path. Windows has a maximum path length of 260 characters by default. If the package's files are deeply nested, this limitation could be exceeded, causing the error when trying to create or access certain files.

C:\\Users\\Nicolas Oliver\\AppData\\Local\\Temp\\pip-install-q11r13x8\\basicsr_8b78bceb4d094ae98f2d8813e165273c\\.eggs\\torch-2.4.1-py3.10-win-amd64.egg\\torch\\include\\ATen\\ops\\_fake_quantize_per_tensor_affine_cachemask_tensor_qparams_compositeexplicitautograd_dispatch.h --> 261 characters... (\\ counted as 1 character)

[dnoliver]

Yes! that was it! Completely forgot about that 260 characters limitation on Windows.

For reference:

1. Open Registry Editor
2. Go to Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
3. Set LongPathsEnbled to 1

Thank you @Nuullll!