PyArrow as optional dependency

this use case makes sense… I agree pyarrow should be an optional dependency (not a package extra).

Removing this as a dependency from the project, we need to:

• remove it from environment.yml file
• run python scripts/generate_pip_deps_from_conda.py to sync change to requirements-dev.txt
• remove it from setup.py file.

Additionally, we should explicitly raise a TypeError when users try to use the pyarrow pandas string format string[pyarrow], which pandera translates to a pandera DataType here: https://github.com/pandera-dev/pandera/blob/master/pandera/engines/pandas_engine.py#L464-L483. Can probably raise the error in the __post_init__ method if storage == "pyarrow".

@jeffzi anything I might have missed here?

Unfortunately serverless does not automatically ignore layer dependencies. You have to list them in the python plugin config.

you are supposed to specify certain dependencies not to be deployed but if it’s combined with Poetry this setting is (silently) ignored.

I also use poetry and I agree the plugin is inconsistent. In some cases, I had to resort to using slimPatterns to force ignore deps:

  pythonRequirements:
    useStaticCache: false
    useDownloadCache: false
    zip: false
    slim: true
    slimPatterns:
      - "botocore*"
      - "cache*"
      - "lxml*"

Another important issue is that layers do actually count for the total size of the deployment package. So splitting packages by layer will not help with that.

Forgot about that 😦 At work I have a dedicated layer for pyarrow. The pyarrow layer strips the *so files your mentioned. It reduces the size, but it’s obviously better to remove pyarrow as a dependency of pandera.