Struggling with fairseq installation

Discussed in https://github.com/pypa/setuptools/discussions/3597

^{Originally posted by voroshilovaleksei September 20, 2022} Hi, I am looking for a help with installing fairseq package. No matter what I’ve tried I always get an error like:

error: Support for editable installs via PEP 660 was recently introduced in setuptools. ERROR: Failed building editable for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects

OS is Windows 10, Anaconda3 I am trying to install fairseq in a fresh env with python 3.7(also tried 3.8 and 3.9) Also tried several setuptools versions (<60.0.0 as well)

What am I missing and what I could try out as well? How do I set env variable SETUPTOOLS_ENABLE_FEATURES=“legacy-editable” ? And if there is a better solution?

Here is a full list of packages installed in that env: conda list # packages in environment at C:\ProgramData\Anaconda3\envs\pytorch:

Name Version Build Channel

pip 22.2.2 pyhd8ed1ab_0 conda-forge python 3.7.0 hc182675_1006 conda-forge python_abi 3.7 2_cp37m conda-forge setuptools 65.3.0 pypi_0 pypi ucrt 10.0.20348.0 h57928b3_0 conda-forge vc 14.2 hb210afc_7 conda-forge vs2015_runtime 14.29.30139 h890b9b1_7 conda-forge wheel 0.37.1 pyhd8ed1ab_0 conda-forge

And a full flow of an installation:

pip install --editable ./ Obtaining file:///C:/Users/avo/Desktop/Kursen/Text%20Mining/Gottbert%20training/fairseq Installing build dependencies … done Checking if build backend supports build_editable … done Getting requirements to build editable … done Installing backend dependencies … done Preparing editable metadata (pyproject.toml) … done Collecting tqdm Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB) Collecting numpy Using cached numpy-1.21.6-cp37-cp37m-win_amd64.whl (14.0 MB) Requirement already satisfied: hydra-core<1.1,>=1.0.7 in c:\users\avo\appdata\roaming\python\python37\site-packages (from fairseq==0.12.2) (1.0.7) Collecting cython Using cached Cython-0.29.32-py2.py3-none-any.whl (986 kB) Collecting cffi Using cached cffi-1.15.1-cp37-cp37m-win_amd64.whl (179 kB) Collecting torchaudio>=0.8.0 Using cached torchaudio-0.12.1-cp37-cp37m-win_amd64.whl (969 kB) Collecting torch Using cached torch-1.12.1-cp37-cp37m-win_amd64.whl (161.9 MB) Collecting bitarray Using cached bitarray-2.6.0-cp37-cp37m-win_amd64.whl (110 kB) Requirement already satisfied: omegaconf<2.1 in c:\users\avo\appdata\roaming\python\python37\site-packages (from fairseq==0.12.2) (2.0.6) Collecting sacrebleu>=1.4.12 Using cached sacrebleu-2.2.1-py3-none-any.whl (116 kB) Collecting regex Using cached regex-2022.9.13-cp37-cp37m-win_amd64.whl (267 kB) Collecting antlr4-python3-runtime==4.8 Using cached antlr4_python3_runtime-4.8-py3-none-any.whl Collecting importlib-resources Using cached importlib_resources-5.9.0-py3-none-any.whl (33 kB) Collecting PyYAML>=5.1.* Using cached PyYAML-6.0-cp37-cp37m-win_amd64.whl (153 kB) Collecting typing-extensions Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB) Collecting colorama Using cached colorama-0.4.5-py2.py3-none-any.whl (16 kB) Collecting tabulate>=0.8.9 Using cached tabulate-0.8.10-py3-none-any.whl (29 kB) Collecting lxml Using cached lxml-4.9.1-cp37-cp37m-win_amd64.whl (3.6 MB) Collecting portalocker Using cached portalocker-2.5.1-py2.py3-none-any.whl (15 kB) Collecting pycparser Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB) Collecting zipp>=3.1.0 Using cached zipp-3.8.1-py3-none-any.whl (5.6 kB) Collecting pywin32>=226 Using cached pywin32-304-cp37-cp37m-win_amd64.whl (12.2 MB) Building wheels for collected packages: fairseq Building editable for fairseq (pyproject.toml) … error error: subprocess-exited-with-error

× Building editable for fairseq (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [97 lines of output] running editable_wheel creating C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info writing C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\PKG-INFO writing dependency_links to C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\dependency_links.txt writing entry points to C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\entry_points.txt writing requirements to C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\requires.txt writing top-level names to C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\top_level.txt writing manifest file ‘C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\SOURCES.txt’ reading manifest file ‘C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\SOURCES.txt’ reading manifest template ‘MANIFEST.in’ adding license file ‘LICENSE’ writing manifest file ‘C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq.egg-info\SOURCES.txt’ creating ‘C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq-0.12.2.dist-info’ adding license file “LICENSE” (matched pattern “LICEN[CS]E*”) creating C:\Users\avo\AppData\Local\Temp\pip-wheel-e7dx4wyf\tmp7rcorala\fairseq-0.12.2.dist-info\WHEEL running build_py running build_ext skipping ‘fairseq/data\data_utils_fast.cpp’ Cython extension (up-to-date) skipping ‘fairseq/data\token_block_utils_fast.cpp’ Cython extension (up-to-date) building ‘fairseq.libbleu’ extension creating C:\Users\avo\AppData\Local\Temp\tmp7l1iu3fg.build-temp\Release creating C:\Users\avo\AppData\Local\Temp\tmp7l1iu3fg.build-temp\Release\fairseq creating C:\Users\avo\AppData\Local\Temp\tmp7l1iu3fg.build-temp\Release\fairseq\clib creating C:\Users\avo\AppData\Local\Temp\tmp7l1iu3fg.build-temp\Release\fairseq\clib\libbleu “C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe” /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\ProgramData\Anaconda3\envs\pytorch\include -IC:\ProgramData\Anaconda3\envs\pytorch\Include “-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE” “-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE” “-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt” “-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um” /EHsc /Tpfairseq/clib/libbleu/libbleu.cpp /FoC:\Users\avo\AppData\Local\Temp\tmp7l1iu3fg.build-temp\Release\fairseq/clib/libbleu/libbleu.obj -std=c++11 -O3 cl : Befehlszeile warning D9002 : Unbekannte Option “-std=c++11” wird ignoriert. cl : Befehlszeile warning D9002 : Unbekannte Option “-O3” wird ignoriert. libbleu.cpp fairseq/clib/libbleu/libbleu.cpp(96): warning C4267: “Argument”: Konvertierung von “size_t” nach “int”, Datenverlust m”glich fairseq/clib/libbleu/libbleu.cpp(102): warning C4267: “Argument”: Konvertierung von “size_t” nach “int”, Datenverlust m”glich “C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe” /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\ProgramData\Anaconda3\envs\pytorch\include -IC:\ProgramData\Anaconda3\envs\pytorch\Include “-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE” “-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE” “-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt” “-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um” /EHsc /Tpfairseq/clib/libbleu/module.cpp /FoC:\Users\avo\AppData\Local\Temp\tmp7l1iu3fg.build-temp\Release\fairseq/clib/libbleu/module.obj -std=c++11 -O3 cl : Befehlszeile warning D9002 : Unbekannte Option “-std=c++11” wird ignoriert. cl : Befehlszeile warning D9002 : Unbekannte Option “-O3” wird ignoriert. module.cpp c:\programdata\anaconda3\envs\pytorch\include\pyconfig.h(215): fatal error C1083: Datei (Include) kann nicht ge”ffnet werden: “basetsd.h”: No such file or directory C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\wheel\bdist_wheel.py:82: RuntimeWarning: Config variable ‘Py_DEBUG’ is unset, Python ABI tag may be incorrect warn=(impl == ‘cp’)): C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\wheel\bdist_wheel.py:87: RuntimeWarning: Config variable ‘WITH_PYMALLOC’ is unset, Python ABI tag may be incorrect sys.version_info < (3, 8)))
C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. setuptools.SetuptoolsDeprecationWarning, Traceback (most recent call last): File “C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools_distutils_msvccompiler.py”, line 421, in compile self.spawn(args) File “C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools_distutils_msvccompiler.py”, line 521, in spawn return super().spawn(cmd, env=env) File “C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools_distutils\ccompiler.py”, line 1007, in spawn spawn(cmd, dry_run=self.dry_run, **kwargs) File “C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools_distutils\spawn.py”, line 71, in spawn “command {!r} failed with exit code {}”.format(cmd, exitcode) distutils.errors.DistutilsExecError: command ‘C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe’ failed with exit code 2

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 140, in run
      self._create_wheel_file(bdist_wheel)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 330, in _create_wheel_file
      files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 261, in _run_build_commands
      self._run_build_subcommands()
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 288, in _run_build_subcommands
      self.run_command(name)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 319, in run_command
      self.distribution.run_command(command)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\dist.py", line 1217, in run_command
      super().run_command(command)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 992, in run_command
      cmd_obj.run()
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\command\build_ext.py", line 84, in run
      _build_ext.run(self)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\Cython\Distutils\old_build_ext.py", line 186, in run
      _build_ext.build_ext.run(self)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 346, in run
      self.build_extensions()
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\Cython\Distutils\old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 466, in build_extensions
      self._build_extensions_serial()
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 492, in _build_extensions_serial
      self.build_extension(ext)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\command\build_ext.py", line 246, in build_extension
      _build_ext.build_extension(self, ext)
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 554, in build_extension
      depends=ext.depends,
    File "C:\Users\avo\AppData\Local\Temp\pip-build-env-7uov75p0\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 423, in compile
      raise CompileError(msg)
  distutils.errors.CompileError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit code 2
  error: Support for editable installs via PEP 660 was recently introduced
  in `setuptools`. If you are seeing this error, please report to:

  https://github.com/pypa/setuptools/issues

  Meanwhile you can try the legacy behavior by setting an
  environment variable and trying to install again:

  SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building editable for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects</div>

Issue Analytics

State:
Created a year ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

abravalhericommented, Sep 20, 2022

Hi @voroshilovaleksei. Yeah compiler errors are very difficult to track (this is the reason why I recommended the environment variable instead of anything fancy: the error is likely related to the compilers, so it does not make much sense to spend energy debugging the “editable install” aspect of it).

This is a tricky one. We can see that the main error here is:

command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit code 2

This might be (or not) a problem with setuptools/distutils (e.g. setuptools/distutils may be missing a flag or creating a wrong command string), but it can be also a problem in the set-up of the compiling tools for Windows.

After a quick search, I found the following results:

https://stackoverflow.com/questions/71242919/pip-install-results-in-this-error-cl-exe-failed-with-exit-code-2
https://github.com/sammchardy/python-binance/issues/148
https://discuss.python.org/t/problem-to-install-pytiff-windows-10-64bits-anaconda-2-1-2-python-3/16230

Can you try having a look if the compiler/libraries are installed? This might be helpful: https://learn.microsoft.com/en-us/visualstudio/python/working-with-c-cpp-python-in-visual-studio?view=vs-2022

There are some related questions that might help when trying to find the problem. Does pip install . work (without the -e)? Does python -m build work (after installing python -m build)?

What happens if you try to create a very minimal example that requires compiling

# pyproject.toml
[build-system]
requires = ["setuptools", "Cython"]
build-backend = "setuptools.build_meta"

[project]
name = "mylib-foo"  # as it would appear on PyPI
version = "0.42"

# foo.pyx
print("hello world")

# setup.py
from setuptools import Extension, setup

setup(
    ext_modules=[
        Extension(
            name="foo",
            sources=["foo.pyx"],
        ),
    ]
)

And build it with python -m build (after installing pip install build)?

You can also try running the following command (with the latest version of pip), to see exactly which compiler commands setuptools is executing and try to see if there is something wrong/missing with them:

pip -v install -e . --config-settings verbose=true

1reaction

abravalhericommented, Sep 20, 2022

Hi @voroshilovaleksei,

And if there is a better solution?

I recommend starting with the environment variable. If the problem persists, the issue is likely to be related with the setup of all the compilers and libraries, which is a completely different beast… Please let me know if setting the environment variable is enough to solve your problem.

How do I set env variable SETUPTOOLS_ENABLE_FEATURES=“legacy-editable” ?

In a Windows environment you have the following options:

Classic CMD terminal: set SETUPTOOLS_ENABLE_FEATURES="legacy-editable" (Ref)
Windows PowerShell: $env:SETUPTOOLS_ENABLE_FEATURES = "legacy-editable" (Ref)

Please note that you have to use the same terminal window to run the build commands (this way of setting environment variables is temporary, if you open a new Window, they are gone and the environment is reset).

Top Results From Across the Web

Cannot install fairseq==0.10.2 with python 3.9 #3535 - GitHub

I don't seem to be able to install fairseq==0.10.2 with python 3.9 on macOS 11.3 $ pip install fairseq==0.10.2 Collecting fairseq==0.10.2 ...

Failing to install fairseq packages - Google Groups

I am a new user in fairseq . I want to implement this model given in this link https://github.com/LiyuanLucasLiu/Transformer-Clinic. But when i ran...

Fairseq Transform model not working (Float can't be cast to long)

I've installed python 3.8, pytorch 1.7, and fairseq 0.10.1, on a new machine, then copied in a script and model from a machine...

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

We leverage the open source toolkit Fairseq (Ott et al., 2019) to investigate the performance of the proposed model in neural machine ...

Retrieval-Augmented Generation (RAG) - ParlAI

Before using RAG, you'll need to make sure that you have installed FAISS; ... Only use this if your machine would struggle to...