[Help] Failed to resolve requirements from PEX environment

See original GitHub issue

I am trying to build a .pex file and then have it run on an EMR cluster. I am getting the following error: Failed to resolve requirements from PEX environment

I am creating a .pex file using the following commands;

export virtual_environment="my_application_env"

python3.7 -m venv $virtual_environment --copies
source $virtual_environment/bin/activate
pip install -U pip

pip download -r requirements.txt \
    --dest ./build/wheelhouse 

pip install pex

pex . -r requirements.txt \
    -o ./$virtual_environment.pex \
    --no-index \
    -f ./build/wheelhouse

rm -rf ./build/wheelhouse
deactivate

Full stack trace:

PEXWarning: PEX_ROOT is configured as /home/.pex but that path is un-writeable, falling back to a temporary P
EX_ROOT of /tmp/tmp_vltgti0 which will hurt performance.
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /mnt/yarn/usercache/hdfs/filecache/1/24059176/connect_cl
oud_base_dq_env.pex.
Needed manylinux_2_26_aarch64-cp-37-cp37m compatible dependencies for:
 1: MarkupSafe>=0.23
    Required by:
      Jinja2 2.11.3
    But this pex had no 'MarkupSafe' distributions.
 2: pyrsistent>=0.14.0
    Required by:
      jsonschema 3.2.0
    But this pex had no 'pyrsistent' distributions.
 3: numpy
    Required by:
      altair 4.1.0
    But this pex had no 'numpy' distributions.
 4: pandas>=0.18
    Required by:
      altair 4.1.0
    But this pex had no 'pandas' distributions.
 5: pyzmq>=13
    Required by:
      jupyter-client 6.2.0
    But this pex had no 'pyzmq' distributions.
 6: tornado>=4.1

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
jsiroiscommented, Apr 21, 2021

Oh, I see you’ve already found the correct target host platform string. Yeah, so you just need to build the PEX file with:

pex . -r requirements.txt \
    -o ./$virtual_environment.pex \
    --no-index \
    -f aarch64-wheelhouse \
    --platform=manylinux_2_26_aarch64-cp-37-cp37m
1reaction
jsiroiscommented, Apr 21, 2021

Aha - yes - exactly. Pex has no magic up its sleeves. To build for arm on an intel host you’ll need two things:

  1. Tell Pex the exact arm64 platform you’ll be running on via --platform
  2. Make sure you have pre-built wheels for the target arm64 platform available for Pex to see when building your PEX file.

For 1, you can find out the correct platform string by running PEX_TOOLS=1 ./my_application_env.pex interpreter -v on the target host (you’ll need to have built the PEX file with --include-tools. You’ll get something like:

$ pex-tools  ./my_application_env.pex interpreter -v
{"path": "/home/jsirois/.venv/pex/bin/python3", "requirement": "CPython==3.9.3", "platform": "manylinux_2_33_x86_64-cp-39-cp39"}

You want to use the “platform” value as the argument to --platform when building your PEX.

For 2, you can use switches similar to those you already have, like --no-index and --find-links pointing to the pre-built wheelhouse.

Read more comments on GitHub >

github_iconTop Results From Across the Web

PEX_PATH transitivity · Issue #1423 · pantsbuild/pex - GitHub
ResolveError: Failed to resolve requirements from PEX environment ... The change in Pex to support creating a venv by using symlinks solves any...
Read more >
pyyaml dependency error while executing with python script in ...
I resolved it by changing env to 3.9 pex requests 'XlsxWriter==1.3.7' 'urllib3==1.26.2' 'PyYAML==5.3.1' ./app -o app.pex --python-shebang '# ...
Read more >
Building .pex files — pex 2.1.119 documentation
This command creates a pex file containing pex and requests, using the console script named “pex”, saving it in ~/bin/pex. At this point,...
Read more >
pex Changelog - pyup.io
This release fixes resolving requirements from a lock using arbitrary equality (``===``). In addition, you can now "inject" runtime environment variables ...
Read more >
Third-party dependencies - Pants build system
dependencies field, Pants will error with a helpful message when building your code with goals like ... Failed to resolve requirements from PEX...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found