[BUG] mlflow after update calls for db migration when no data has changed

Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

System information

Have I written custom code (as opposed to using a stock example script provided in MLflow):yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu:bionic, ubuntu:focal
MLflow installed from (source or binary): binary
MLflow version (run mlflow --version): 1.3.0, 1.8.0
Python version: 3.6.4, 3.8.2
Exact command to reproduce: -

Describe the problem

I upgraded mlflow server version in our mirekphd/ml-mlflow docker container from 1.3.0 (container tag 20191024) to 1.8.0 (tag 20200611). The new mlflow version fails to start due to apparent need for db schema migration, even though nothing has changed and rollback of mlflow to previous version 1.3.0 restores normal operation

Code to reproduce issue

buggy: docker pull mirekphd/ml-mlflow:20200611 vs. normal: docker pull mirekphd/ml-mlflow:20191024

Other info / logs

2020/06/11 11:26:28 ERROR mlflow.cli: Error initializing backend store
2020/06/11 11:26:28 ERROR mlflow.cli: Detected out-of-date database schema (found version 89d4b8295536, but expected 0a8213491aaa). Take a backup of your database, then run 'mlflow db upgrade <database_uri>' to migrate your database to the latest schema. NOTE: schema migration may result in database downtime - please consult your database's documentation for more detail.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/mlflow/cli.py", line 286, in server
    initialize_backend_stores(backend_store_uri, default_artifact_root)
  File "/usr/local/lib/python3.8/dist-packages/mlflow/server/handlers.py", line 101, in initialize_backend_stores
    _get_tracking_store(backend_store_uri, default_artifact_root)
  File "/usr/local/lib/python3.8/dist-packages/mlflow/server/handlers.py", line 87, in _get_tracking_store
    _tracking_store = _tracking_store_registry.get_store(store_uri, artifact_root)
  File "/usr/local/lib/python3.8/dist-packages/mlflow/tracking/_tracking_service/registry.py", line 37, in get_store
    return builder(store_uri=store_uri, artifact_uri=artifact_uri)
  File "/usr/local/lib/python3.8/dist-packages/mlflow/server/handlers.py", line 60, in _get_sqlalchemy_store
    return SqlAlchemyStore(store_uri, artifact_uri)
  File "/usr/local/lib/python3.8/dist-packages/mlflow/store/tracking/sqlalchemy_store.py", line 103, in __init__
    mlflow.store.db.utils._verify_schema(self.engine)
  File "/usr/local/lib/python3.8/dist-packages/mlflow/store/db/utils.py", line 50, in _verify_schema
    raise MlflowException(
mlflow.exceptions.MlflowException: Detected out-of-date database schema (found version 89d4b8295536, but expected 0a8213491aaa). Take a backup of your database, then run 'mlflow db upgrade <database_uri>' to migrate your database to the latest schema. NOTE: schema migration may result in database downtime - please consult your database's documentation for more detail.

What component(s), interfaces, languages, and integrations does this bug affect?

Components

area/tracking: Tracking Service, tracking client APIs, autologging

Interface

area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry

Issue Analytics

State:
Created 3 years ago
Comments:8

Top GitHub Comments

3reactions

mirekphdcommented, Aug 25, 2020

So I finally modified the container template to replace the entrypoint so as to run mlflow db upgrade <sqlalchemy_db_conn_str> instead of mlflow server <params>, using the desired new version of mlflow==1.10.0,

It printed this message (which is apparently normal):

2020/08/25 10:07:33 INFO mlflow.store.db.utils: Updating database tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

… and then I was eventually able to run the standard MLflow container (mirekphd/ml-mlflow:20200825), and display all previously archived metrics and params data (served by MariaDB migrated earlier independently to 10.5) in this new mlflow version.

Caution: please be aware that I performed database upgrade independently (using dedicated tool, mysql_upgrade) before trying this migration using mlflow db upgrade, so if you skip the former step, a new database version will most likely refuse to recognize existing data committed using an older database version.

0reactions

mirekphdcommented, Jun 17, 2022

@tanlin2013 an example of an arguably safer MariaDB auto-migration, preceded by a backup of system tables (using docker_mariadb_backup_system) and a check if such upgrade would be needed (performed in _check_if_upgrade_is_needed), can be found in the docker_mariadb_upgrade function in the official MariaDB container.