poetry add scikit-learn does not work on m1 mac

See original GitHub issue
  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • OS version and name: 2021 MacBook Pro with Apple M1 Pro - macOS Monterey 12.3.1
  • Poetry version: 1.1.13
  • Link of a Gist with the contents of your pyproject.toml file:
[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["SamEdwardes <edwardes.s@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.27.1"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Issue

I am unable to install scikit-learn using poetry on an m1 macbook pro. Everything seems to work:

$ poetry init --no-interaction --name example
$ poetry env use python3.10
$ poetry add requests
$ poetry run python
Python 3.10.4 (main, May 16 2022, 17:43:57) [Clang 13.1.6 (clang-1316.0.21.2.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Until I attempt scikit-learn:

$ poetry add -vvv scikit-learn
expand for output 
Using virtualenv: /Users/samedwardes/tmp/poetry-issue/.venv
PyPI: No release information found for scikit-learn-0.14a1, skipping
PyPI: 45 packages found for scikit-learn *
Using version ^1.1.0 for scikit-learn

Updating dependencies
Resolving dependencies...
   1: fact: example is 0.1.0
   1: derived: example
   1: fact: example depends on requests (^2.27.1)
   1: fact: example depends on scikit-learn (^1.1.0)
   1: selecting example (0.1.0)
   1: derived: scikit-learn (>=1.1.0,<2.0.0)
   1: derived: requests (>=2.27.1,<3.0.0)
PyPI: No release information found for scikit-learn-0.14a1, skipping
PyPI: 1 packages found for scikit-learn >=1.1.0,<2.0.0
   1: fact: scikit-learn (1.1.0) depends on numpy (>=1.17.3)
   1: fact: scikit-learn (1.1.0) depends on scipy (>=1.3.2)
   1: fact: scikit-learn (1.1.0) depends on joblib (>=1.0.0)
   1: fact: scikit-learn (1.1.0) depends on threadpoolctl (>=2.0.0)
   1: selecting scikit-learn (1.1.0)
   1: derived: threadpoolctl (>=2.0.0)
   1: derived: joblib (>=1.0.0)
   1: derived: scipy (>=1.3.2)
   1: derived: numpy (>=1.17.3)
   1: fact: requests (2.27.1) depends on urllib3 (>=1.21.1,<1.27)
   1: fact: requests (2.27.1) depends on certifi (>=2017.4.17)
   1: fact: requests (2.27.1) depends on charset-normalizer (>=2.0.0,<2.1.0)
   1: fact: requests (2.27.1) depends on idna (>=2.5,<4)
   1: selecting requests (2.27.1)
   1: derived: idna (>=2.5,<4)
   1: derived: charset-normalizer (>=2.0.0,<2.1.0)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<1.27)
   1: selecting threadpoolctl (3.1.0)
   1: selecting joblib (1.1.0)
   1: fact: scipy (1.6.1) depends on numpy (>=1.16.5)
   1: selecting scipy (1.6.1)
   1: selecting numpy (1.22.3)
   1: selecting certifi (2021.10.8)
   1: selecting urllib3 (1.26.9)
   1: selecting idna (3.3)
   1: selecting charset-normalizer (2.0.12)
   1: Version solving took 0.021 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 2 installs, 0 updates, 0 removals, 8 skipped

  • Installing numpy (1.22.3): Pending...
  • Installing numpy (1.22.3): Skipped for the following reason: Already installed
  • Installing certifi (2021.10.8): Pending...
  • Installing certifi (2021.10.8): Skipped for the following reason: Already installed
  • Installing charset-normalizer (2.0.12): Pending...
  • Installing charset-normalizer (2.0.12): Skipped for the following reason: Already installed
  • Installing idna (3.3): Pending...
  • Installing idna (3.3): Skipped for the following reason: Already installed
  • Installing joblib (1.1.0): Pending...
  • Installing joblib (1.1.0): Skipped for the following reason: Already installed
  • Installing scipy (1.6.1): Pending...
  • Installing scipy (1.6.1): Installing...
  • Installing scipy (1.6.1): Failed

  EnvCommandError

  Command ['/Users/samedwardes/tmp/poetry-issue/.venv/bin/pip', 'install', '--no-deps', 'file:///Users/samedwardes/Library/Caches/pypoetry/artifacts/bc/74/e2/a22451059efb69c32be2d51a7727b1133375122dd7564e8a8e9df4f1cd/scipy-1.6.1.tar.gz'] errored with the following return code 1, and output:
  Processing /Users/samedwardes/Library/Caches/pypoetry/artifacts/bc/74/e2/a22451059efb69c32be2d51a7727b1133375122dd7564e8a8e9df4f1cd/scipy-1.6.1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error

    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [100 lines of output]
        setup.py:461: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /private/var/folders/v9/fyndz9rs2hs34xwbs1rtmqtm0000gn/T/pip-modern-metadata-h45cphql'), proceeding with generating Cython sources and expanding templates
          warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
        Running from SciPy source directory.
        INFO: lapack_opt_info:
        INFO: lapack_armpl_info:
        INFO: customize UnixCCompiler
        INFO:   libraries armpl_lp64_mp not found in ['/Users/samedwardes/tmp/poetry-issue/.venv/lib', '/usr/lib']
        INFO:   NOT AVAILABLE
        INFO:
        INFO: lapack_mkl_info:
        INFO:   libraries mkl_rt not found in ['/Users/samedwardes/tmp/poetry-issue/.venv/lib', '/usr/lib']
        INFO:   NOT AVAILABLE
        INFO:
        INFO: openblas_lapack_info:
        INFO:   libraries openblas not found in ['/Users/samedwardes/tmp/poetry-issue/.venv/lib', '/usr/lib']
        INFO:   NOT AVAILABLE
        INFO:
        INFO: openblas_clapack_info:
        INFO:   libraries openblas,lapack not found in ['/Users/samedwardes/tmp/poetry-issue/.venv/lib', '/usr/lib']
        INFO:   NOT AVAILABLE
        INFO:
        INFO: flame_info:
        INFO:   libraries flame not found in ['/Users/samedwardes/tmp/poetry-issue/.venv/lib', '/usr/lib']
        INFO:   NOT AVAILABLE
        INFO:
        INFO: accelerate_info:
        INFO:   NOT AVAILABLE
        INFO:
        INFO: atlas_3_10_threads_info:
        INFO: Setting PTATLAS=ATLAS
        INFO:   libraries tatlas,tatlas not found in /Users/samedwardes/tmp/poetry-issue/.venv/lib
        INFO:   libraries tatlas,tatlas not found in /usr/lib
        INFO: <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
        INFO:   NOT AVAILABLE
        INFO:
        INFO: atlas_3_10_info:
        INFO:   libraries satlas,satlas not found in /Users/samedwardes/tmp/poetry-issue/.venv/lib
        INFO:   libraries satlas,satlas not found in /usr/lib
        INFO: <class 'numpy.distutils.system_info.atlas_3_10_info'>
        INFO:   NOT AVAILABLE
        INFO:
        INFO: atlas_threads_info:
        INFO: Setting PTATLAS=ATLAS
        INFO:   libraries ptf77blas,ptcblas,atlas not found in /Users/samedwardes/tmp/poetry-issue/.venv/lib
        INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/lib
        INFO: <class 'numpy.distutils.system_info.atlas_threads_info'>
        INFO:   NOT AVAILABLE
        INFO:
        INFO: atlas_info:
        INFO:   libraries f77blas,cblas,atlas not found in /Users/samedwardes/tmp/poetry-issue/.venv/lib
        INFO:   libraries f77blas,cblas,atlas not found in /usr/lib
        INFO: <class 'numpy.distutils.system_info.atlas_info'>
        INFO:   NOT AVAILABLE
        INFO:
        INFO: lapack_info:
        INFO:   libraries lapack not found in ['/Users/samedwardes/tmp/poetry-issue/.venv/lib', '/usr/lib']
        INFO:   NOT AVAILABLE
        INFO:
        /private/var/folders/v9/fyndz9rs2hs34xwbs1rtmqtm0000gn/T/pip-build-env-h61s4zsi/overlay/lib/python3.10/site-packages/numpy/distutils/system_info.py:1902: UserWarning:
            Lapack (http://www.netlib.org/lapack/) libraries not found.
            Directories to search for the libraries can be specified in the
            numpy/distutils/site.cfg file (section [lapack]) or by setting
            the LAPACK environment variable.
          return getattr(self, '_calc_info_{}'.format(name))()
        INFO: lapack_src_info:
        INFO:   NOT AVAILABLE
        INFO:
        /private/var/folders/v9/fyndz9rs2hs34xwbs1rtmqtm0000gn/T/pip-build-env-h61s4zsi/overlay/lib/python3.10/site-packages/numpy/distutils/system_info.py:1902: UserWarning:
            Lapack (http://www.netlib.org/lapack/) sources not found.
            Directories to search for the sources can be specified in the
            numpy/distutils/site.cfg file (section [lapack_src]) or by setting
            the LAPACK_SRC environment variable.
          return getattr(self, '_calc_info_{}'.format(name))()
        INFO:   NOT AVAILABLE
        INFO:
        Traceback (most recent call last):
          File "/Users/samedwardes/tmp/poetry-issue/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
            main()
          File "/Users/samedwardes/tmp/poetry-issue/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/Users/samedwardes/tmp/poetry-issue/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
            return hook(metadata_directory, config_settings)
          File "/private/var/folders/v9/fyndz9rs2hs34xwbs1rtmqtm0000gn/T/pip-build-env-h61s4zsi/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
            self.run_setup()
          File "/private/var/folders/v9/fyndz9rs2hs34xwbs1rtmqtm0000gn/T/pip-build-env-h61s4zsi/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 253, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/private/var/folders/v9/fyndz9rs2hs34xwbs1rtmqtm0000gn/T/pip-build-env-h61s4zsi/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 145, in run_setup
            exec(compile(code, __file__, 'exec'), locals())
          File "setup.py", line 588, in <module>
            setup_package()
          File "setup.py", line 584, in setup_package
            setup(**metadata)
          File "/private/var/folders/v9/fyndz9rs2hs34xwbs1rtmqtm0000gn/T/pip-build-env-h61s4zsi/overlay/lib/python3.10/site-packages/numpy/distutils/core.py", line 135, in setup
            config = configuration()
          File "setup.py", line 499, in configuration
            raise NotFoundError(msg)
        numpy.distutils.system_info.NotFoundError: No BLAS/LAPACK libraries found. Note: Accelerate is no longer supported.
        To build Scipy from sources, BLAS & LAPACK libraries need to be installed.
        See site.cfg.example in the Scipy source directory and
        https://docs.scipy.org/doc/scipy/reference/building/index.html for details.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed

  × Encountered error while generating package metadata.
  ╰─> See above for output.

  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.
  WARNING: You are using pip version 22.0.4; however, version 22.1 is available.
  You should consider upgrading via the '/Users/samedwardes/tmp/poetry-issue/.venv/bin/python -m pip install --upgrade pip' command.


  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:1195 in _run
      1191│                 output = subprocess.check_output(
      1192│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1193│                 )
      1194│         except CalledProcessError as e:
    → 1195│             raise EnvCommandError(e, input=input_)
      1196│
      1197│         return decode(output)
      1198│
      1199│     def execute(self, bin, *args, **kwargs):

  • Installing threadpoolctl (3.1.0): Pending...
  • Installing threadpoolctl (3.1.0): Skipped for the following reason: Already installed
  • Installing urllib3 (1.26.9): Pending...
  • Installing urllib3 (1.26.9): Skipped for the following reason: Already installed

Failed to add packages, reverting the pyproject.toml file to its original content.

There is known issue with scikit-learn on the M1 mac (https://scikit-learn.org/stable/install.html#installing-on-apple-silicon-m1-hardware):

The recently introduced macos/arm64 platform (sometimes also known as macos/aarch64) requires the open source community to upgrade the build configuration and automation to properly support it.

At the time of writing (January 2021), the only way to get a working installation of scikit-learn on this hardware is to install scikit-learn and its dependencies from the conda-forge distribution, for instance using the miniforge installers:

https://github.com/conda-forge/miniforge

The following issue tracks progress on making it possible to install scikit-learn from PyPI with pip:

https://github.com/scikit-learn/scikit-learn/issues/19137

But it looks like this was fixed in a recent PR (https://github.com/scikit-learn/scikit-learn/pull/21827). I also confirmed that I was able to install scikit-learn via pip on my m1.

A few more details that may help:

  • Using a python installed via pyenv (e.g. pyenv install 3.10.4)

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:12 (7 by maintainers)

github_iconTop GitHub Comments

4reactions
SamEdwardescommented, May 19, 2022

Thank you @branchvincent. I did see your comment. I though I tried setting python = ">=3.9,<3.11" and it still did not work unless I installed scipy first. That was incorrect! I think I just got mixed up in all of the many variations I tried.

To close the loop and to confirm for future readers the following works:

$ poetry init --no-interaction --name example --python ">=3.9,<3.11"
$ poetry add scikit-learn

This command will guide you through creating your pyproject.toml config.


You can specify a package in the following forms:
  - A single name (requests)
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)



Creating virtualenv example in /Users/samedwardes/tmp/poetry-test/test-8/.venv
Using version ^1.1.1 for scikit-learn

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 5 installs, 0 updates, 0 removals

  • Installing numpy (1.22.3)
  • Installing joblib (1.1.0)
  • Installing scipy (1.8.1)
  • Installing threadpoolctl (3.1.0)
  • Installing scikit-learn (1.1.1)

Thanks for the help everyone!

2reactions
branchvincentcommented, May 19, 2022

@SamEdwardes did you see my comment? https://github.com/python-poetry/poetry/issues/5628#issuecomment-1129422709

the issue here is python = "^3.9" is filtering out the latest scipy. Can you verify that the following works for you:

$ cd "$(mktemp -d)"
$ poetry init --no-interaction --name example --python '>=3.9,<3.11'
$ poetry add scikit-learn
Package operations: 5 installs, 0 updates, 0 removals

  • Installing numpy (1.22.3)
  • Installing joblib (1.1.0)
  • Installing scipy (1.8.1)
  • Installing threadpoolctl (3.1.0)
  • Installing scikit-learn (1.1.1)
Read more comments on GitHub >

github_iconTop Results From Across the Web

Installing scipy and scikit-learn on apple m1 - Stack Overflow
UPDATE: scikit-learn now works via pip ✓. Just first brew install openblas - it has instructions for different processors (wikipedia)
Read more >
[FIXED] Installing scipy and scikit-learn on apple m1
However when I try to install scipy or scikit-learn via pip this error ... [FIXED] Visual studio code: "matplotlib" is not accessed Pylance....
Read more >
Installing Ray — Ray 2.2.0 - the Ray documentation
Ray has experimental support for machines running Apple Silicon (such as M1 macs). To get started: Install miniforge. ... Ensure you're using the...
Read more >
Install Scikit Learn on Apple Silicon M1 - Sukhpal Saini
I spent hours looking for the right answer. Here's the step-by-step. ... You should now be able to do which conda to see...
Read more >
Current situation of library support for M1 : r/Python - Reddit
M1 user here. Current situation is following: docker runs with no problems. numpy, sklearn, etc already have wheels so pip works.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found