[BUG] `AttributeError: 'PosixPath' object has no attribute 'read_text'` with setuptools 60.9.0+ and `pathlib`

setuptools version

setuptools==62.1.0 (however the earliest affected version is setuptools==60.9.0)

Python version

Python 3.9 (though 3.7 and 3.8 are also affected, plus 3.10 is affected but gives a different error message)

OS

Docker image python:3.9.12 which is Debian 11 (it also occurs on Ubuntu on Heroku)

Additional environment information

mrio-common-metadata==0.2

(which indirectly pulls in pathlib==1.0.1)

Description

Between setuptools 60.8.2 and 60.9.0, a previously working project now encounters an error during the python setup.py egg_info phase of pip install:

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ymj9drfq/pathlib_99a5e0894a554de3b2297b3e6602e3f7/setup.py", line 6, in <module>
          setup(
        File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 109, in setup
          _setup_distribution = dist = klass(attrs)
        File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 460, in __init__
          for ep in metadata.entry_points(group='distutils.setup_keywords'):
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 999, in entry_points
          return SelectableGroups.load(eps).select(**params)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 449, in load
          ordered = sorted(eps, key=by_group)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 997, in <genexpr>
          dist.entry_points for dist in unique(distributions())
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 609, in entry_points
          return EntryPoints._from_text_for(self.read_text('entry_points.txt'), self)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 917, in read_text
          return self._path.joinpath(filename).read_text(encoding='utf-8')
      AttributeError: 'PosixPath' object has no attribute 'read_text'

The changes between those two setuptools versions are: https://github.com/pypa/setuptools/compare/v60.8.2...v60.9.0

Under Python 3.7/3.8 the failure is the same as seen on Python 3.9 above.

However under Python 3.10, the error changes to:

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Downgrading to setuptools==60.8.2 resolves the issue.

It seems that the pathlib package in the environment is being used by setuptools instead of the module in the stdlib. This should not happen, since the stdlib has higher precedence on sys.path than site-packages.

See also previously filed #3132. I’ve filed this separately, since:

• that issue is closed (I believe incorrectly, due to a misunderstanding about sys.path precedence)
• the steps to reproduce here are slightly different (a real world case encountered in a customer app)
• this issue has more details than the other one

…however feel free to dupe over there and reopen that issue if preferred 😃

Expected behavior

Either:

• Installing mrio_common_metadata==0.2 continues to work as it did for setuptools <= 60.8.2.
• Or, as a major version release (not a minor version bump), setuptools causes this scenario to be an error with an easy to debug error message.

How to Reproduce

For the Python 3.9 repro:

1. docker run --rm -it python:3.9.12 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'

For the Python 3.10 alternative error message variant:

1. docker run --rm -it python:3.10.4 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'

Output

Python 3.9:

$ docker run --rm -it python:3.9.12 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'
...
Successfully installed setuptools-62.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting mrio_common_metadata==0.2
  Downloading mrio_common_metadata-0.2.tar.gz (12 kB)
  Preparing metadata (setup.py) ... done
Collecting pyxlsb
  Downloading pyxlsb-1.0.9-py2.py3-none-any.whl (23 kB)
Collecting pathlib
  Downloading pathlib-1.0.1.tar.gz (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 KB 24.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ymj9drfq/pathlib_99a5e0894a554de3b2297b3e6602e3f7/setup.py", line 6, in <module>
          setup(
        File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 109, in setup
          _setup_distribution = dist = klass(attrs)
        File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 460, in __init__
          for ep in metadata.entry_points(group='distutils.setup_keywords'):
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 999, in entry_points
          return SelectableGroups.load(eps).select(**params)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 449, in load
          ordered = sorted(eps, key=by_group)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 997, in <genexpr>
          dist.entry_points for dist in unique(distributions())
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 609, in entry_points
          return EntryPoints._from_text_for(self.read_text('entry_points.txt'), self)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 917, in read_text
          return self._path.joinpath(filename).read_text(encoding='utf-8')
      AttributeError: 'PosixPath' object has no attribute 'read_text'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Python 3.10:

$ docker run --rm -it python:3.10.4 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'
...
Successfully installed setuptools-62.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting mrio_common_metadata==0.2
  Downloading mrio_common_metadata-0.2.tar.gz (12 kB)
  Preparing metadata (setup.py) ... done
Collecting pyxlsb
  Downloading pyxlsb-1.0.9-py2.py3-none-any.whl (23 kB)
Collecting pathlib
  Downloading pathlib-1.0.1.tar.gz (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 KB 24.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.