Memory leak after updating to Pydantic v1.9.0

See original GitHub issue

Checks

  • I added a descriptive title to this issue
  • I have searched (google, github) for similar issues and couldn’t find anything
  • I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

$ python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.9.0
            pydantic compiled: True
                 install path: /opt/pydist/.venv/lib/python3.10/site-packages/pydantic
               python version: 3.10.1 (main, Dec 21 2021, 09:01:08) [GCC 10.2.1 20210110]
                     platform: Linux-5.10.93-x86_64-with-glibc2.31
     optional deps. installed: ['dotenv', 'typing-extensions']

As a heads-up we are seeing a memory leak in production after upgrading to Pydantic version 1.9.0. Our application is a large FastAPI based API service (fastapi 0.73.0, uvicorn 0.17.4). The application uses Pydantic BaseModels, GenericModels and pydantic.dataclasses.dataclasses for FastAPI response/request data models. It also uses Pydantic models for some internal data validators. We deployed a single change to production, which changed the version of Pydantic from 1.8.2 to 1.9.0. After that we started to observe abnormal memory usage. The memory usage increased until the version upgrade got reverted back to the older 1.8.2 version (no other change again). After deploying the revert, the memory usage patterns went back to normal.

Unfortunately I haven’t been able to reproduce the issue locally (it is an effort to simulate the prod-like load to get the leak to show up). So I can not yet add further information from isolating it or information from eg. memory analyzers. I tried to also look into the changelog and diff but its so big release that I can not spot anything obvious that may cause this. I have attached below a picture from our production metrics about the abnormal behaviour after using v1.9.0. We have never observed such a memory issue until now when upgrading to v1.9.0.

Thank you all for the awesome work and the great library! I wish I would have more information about the leak for the ticket.

Näyttökuva 2022-2-18 kello 8 48 24 Above shows the different deploys in colors. The orange one is where the Pydantic 1.9.0 update was running until it was reverted due to the memory issue. We couldn't have run it any longer as it was close at hitting the Kubernetes pod memory limits at that point.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:2
  • Comments:27 (15 by maintainers)

github_iconTop GitHub Comments

3reactions
samuelcolvincommented, May 14, 2022

Well after hours of hunting, the good news is I think I’ve found a memory leak, the bad news is it’s in python and I can’t see how it’s specific to v1.9.

https://github.com/python/cpython/issues/92810

😞

2reactions
samuelcolvincommented, May 16, 2022

Yes I think so.

Also:

  • it’s probably not used by that many people, which might explain why not everyone sees this
  • It definitely changed in v1.9
  • there’s an obvious way for memory to build up via _assigned_parameters

I think I’ll put a limit on the size of those two cache dicts just in case.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Memory leak after updating to Pydantic v1.9.0 #3829 - GitHub
As a heads-up we are seeing a memory leak in production after upgrading to Pydantic version 1.9.0. Our application is a large FastAPI...
Read more >
Changelog - pydantic
Revert Breaking Change: v1.9.1 introduced a breaking change where model fields were ... models when the type is not a model, may also...
Read more >
pydantic - Browse /v1.9.1 at SourceForge.net
Data parsing and validation using Python type hints.
Read more >
Pydantic 2 rewritten in Rust was merged : r/Python - Reddit
Last update was ~17x faster for most operations. ... Rust doesn't save you from deadlocks, memory leaks, or race conditions ... 9 days...
Read more >
ChangeLog - ftp
(#521) Update pydantic hook to include dotenv optional dependency. ... dnsc_conf_set memory leak * developer visible fixes ++++ mdevctl: - Update to version ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found