Backend failed to start up, Error: The migration directory is corrupt, the following files are missing

Expected Behavior

The backend should start up successfully

Actual Behavior

An error occurs.

[1] Backend failed to start up, Error: The migration directory is corrupt, the following files are missing: 20200511113813_init.js, 20200520140700_location_update_log_table.js, 20200527114117_location_update_log_latest_view.js, 20200702153613_entities.js, 20200721115244_location_update_log_latest_deduplicate.js, 20200805163904_location_update_log_duplication_fix.js, 20200807120600_entitySearch.js, 20200809202832_add_bootstrap_location.js, 20200923104503_case_insensitivity.js, 20201005122705_add_entity_full_name.js, 20201006130744_entity_data_column.js, 20201006203131_entity_remove_redundant_columns.js, 20201007201501_index_entity_search.js, 20201019130742_add_relations_table.js, 20201123205611_relations_table_uniq.js, 20201210185851_fk_index.js, 20201230103504_update_log_varchar.js, 20210209121210_locations_fk_index.js, 20210302150147_refresh_state.js, 20210622104022_refresh_state_location_key.js, 20210813143113_add_refresh_state_hash.js, 20210925102509_add_refresh_state_input_hash.js, 20220116144621_remove_legacy.js, 20220222164811_reprocess_for_relation_refs.js

Steps to Reproduce

Configure Postgres connection as such:

  database:
    # config options: https://node-postgres.com/api/client
    client: pg
    connection:
      connectionString: 'postgres://backstage:secret@machinename/backstage'
      # host: machinename
      # port: 5432
      # user: backstage
      # database: 'backstage'
      # password: secret
      # https://node-postgres.com/features/ssl
      ssl: # see https://www.postgresql.org/docs/current/libpq-ssl.html Table 33.1. SSL Mode Descriptions (e.g. require)
        rejectUnauthorized: false

Run yarn dev
See error

Context

I’ve been trying to get my Backstage instance connected to a Postgres database for a few days now but I always end up back in this same spot.

Things I’ve tried:

Clean install/redo the changes I need
Deleted all node_modules folders and reinstalled
Dropped and recreated all databases related to Backstage
Manually removed rows from migrations table

Also, I am using the connectionString configuration property because I need to specify the maintenance database. We cannot get read/write access to the default postgres database per our account security policies, and the database configuration property is not adhered to it seems.

I found these files in the catalog-backend which appears to be exactly what the error message is pointing to.

Any help would be greatly appreciated!

Your Environment

Running Ubuntu 20

Browser Information: n/a
Output of yarn backstage-cli info:

OS:   Linux 5.4.0-107-generic - linux/x64
node: v16.15.0
yarn: 1.22.18
cli:  0.17.0 (installed)

Dependencies:
  @backstage/app-defaults                        1.0.1
  @backstage/backend-common                      0.13.2
  @backstage/backend-tasks                       0.3.0
  @backstage/catalog-client                      1.0.1
  @backstage/catalog-model                       1.0.1
  @backstage/cli-common                          0.1.8
  @backstage/cli                                 0.17.0
  @backstage/config-loader                       1.1.0
  @backstage/config                              1.0.0
  @backstage/core-app-api                        1.0.1
  @backstage/core-components                     0.9.3
  @backstage/core-plugin-api                     1.0.1
  @backstage/errors                              1.0.0
  @backstage/integration-react                   1.0.1
  @backstage/integration                         1.1.0
  @backstage/plugin-api-docs                     0.8.4
  @backstage/plugin-app-backend                  0.3.31
  @backstage/plugin-auth-backend                 0.13.0
  @backstage/plugin-auth-node                    0.2.0
  @backstage/plugin-azure-devops-backend         0.3.10
  @backstage/plugin-azure-devops-common          0.2.2
  @backstage/plugin-azure-devops                 0.1.20
  @backstage/plugin-catalog-backend-module-azure 0.1.2
  @backstage/plugin-catalog-backend              1.1.1
  @backstage/plugin-catalog-common               1.0.1
  @backstage/plugin-catalog-graph                0.2.16
  @backstage/plugin-catalog-import               0.8.7
  @backstage/plugin-catalog-react                1.0.1
  @backstage/plugin-catalog                      1.1.0
  @backstage/plugin-github-actions               0.5.4
  @backstage/plugin-org                          0.5.4
  @backstage/plugin-permission-common            0.6.0
  @backstage/plugin-permission-node              0.6.0
  @backstage/plugin-permission-react             0.4.0
  @backstage/plugin-proxy-backend                0.2.25
  @backstage/plugin-scaffolder-backend           1.1.0
  @backstage/plugin-scaffolder-common            1.0.1
  @backstage/plugin-scaffolder                   1.1.0
  @backstage/plugin-search-backend-node          0.6.0
  @backstage/plugin-search-backend               0.5.1
  @backstage/plugin-search-common                0.3.3
  @backstage/plugin-search-react                 0.1.0
  @backstage/plugin-search                       0.8.0
  @backstage/plugin-tech-radar                   0.5.11
  @backstage/plugin-techdocs-backend             1.1.0
  @backstage/plugin-techdocs-node                1.1.0
  @backstage/plugin-techdocs-react               0.1.0
  @backstage/plugin-techdocs                     1.1.0
  @backstage/plugin-user-settings                0.4.3
  @backstage/release-manifests                   0.0.2
  @backstage/search-common                       0.3.3
  @backstage/test-utils                          1.0.1
  @backstage/theme                               0.2.15
  @backstage/types                               1.0.0
  @backstage/version-bridge                      1.0.1

Issue Analytics

State:
Created a year ago
Comments:13 (4 by maintainers)

Top GitHub Comments

2reactions

JoostSaanencommented, Oct 24, 2022

@arhill05 I’m stuck on this issue. Can you please provide some more information. I’m currently have the following config:

app:
  # Should be the same as backend.baseUrl when using the `app-backend` plugin.
  baseUrl: https://example.herokuapp.com
  title: example

backend:
  # Note that the baseUrl should be the URL that the browser and other clients
  # should use when communicating with the backend, i.e. it needs to be
  # reachable not just from within the backend host, but from all of your
  # callers. When its value is "http://localhost:7007", it's strictly private
  # and can't be reached by others.
  baseUrl: https://example.herokuapp.com
  listen:
    port:
      $env: PORT

  # config options: https://node-postgres.com/api/client
  database:
    client: pg
    ensureExists: false
    pluginDivisionMode: 'schema'
    connection: ${DATABASE_URL}
    ssl: true
    knexConfig:
      searchPath: ['knex', 'public']
    plugin:
      auth:
        database:
          $env: DATABASE_URL
      catalog:
        database:
          $env: DATABASE_URL
techdocs:
  builder: 'local' # Alternatives - 'external'
  generator:
    runIn: 'docker' # Alternatives - 'local'
  publisher:
    type: 'local' # Alternatives - 'googleGcs' or 'awsS3'. Read documentation for using alternatives.


catalog:
  # Overrides the default list locations from app-config.yaml as these contain example data.
  # See https://backstage.io/docs/features/software-catalog/software-catalog-overview#adding-components-to-the-catalog for more details
  # on how to get entities into the catalog.
  locations: []

I’m still running in the error:

2022-10-24T19:39:55.297434+00:00 app[web.1]: {"level":"info","message":"Loaded config from app-config.yaml, app-config.production.yaml","service":"backstage"}
2022-10-24T19:39:55.307329+00:00 app[web.1]: {"level":"info","message":"1 secret found in the config which will be redacted","service":"backstage"}
2022-10-24T19:39:55.322286+00:00 app[web.1]: {"level":"info","message":"Created UrlReader predicateMux{readers=azure{host=dev.azure.com,authed=false},bitbucketCloud{host=bitbucket.org,authed=false},github{host=github.com,authed=false},gitlab{host=gitlab.com,authed=false},awsS3{host=amazonaws.com,authed=false},fetch{}","service":"backstage"}
2022-10-24T19:39:55.347242+00:00 app[web.1]: {"level":"info","message":"Performing database migration","plugin":"catalog","service":"backstage","type":"plugin"}
2022-10-24T19:39:55.463702+00:00 app[web.1]: Backend failed to start up Error: The migration directory is corrupt, the following files are missing: 20200511113813_init.js, 20200520140700_location_update_log_table.js, 20200527114117_location_update_log_latest_view.js, 20200702153613_entities.js, 20200721115244_location_update_log_latest_deduplicate.js, 20200805163904_location_update_log_duplication_fix.js, 20200807120600_entitySearch.js, 20200809202832_add_bootstrap_location.js, 20200923104503_case_insensitivity.js, 20201005122705_add_entity_full_name.js, 20201006130744_entity_data_column.js, 20201006203131_entity_remove_redundant_columns.js, 20201007201501_index_entity_search.js, 20201019130742_add_relations_table.js, 20201123205611_relations_table_uniq.js, 20201210185851_fk_index.js, 20201230103504_update_log_varchar.js, 20210209121210_locations_fk_index.js, 20210302150147_refresh_state.js, 20210622104022_refresh_state_location_key.js, 20210813143113_add_refresh_state_hash.js, 20210925102509_add_refresh_state_input_hash.js, 20220116144621_remove_legacy.js, 20220222164811_reprocess_for_relation_refs.js, 20220616202842_refresh_keys.js
2022-10-24T19:39:55.463708+00:00 app[web.1]: at validateMigrationList (/app/node_modules/knex/lib/migrations/migrate/Migrator.js:566:11)
2022-10-24T19:39:55.463708+00:00 app[web.1]: at Migrator.latest (/app/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)

1reaction

kwentcommented, Dec 8, 2022

Hi, could we have some help and guidance from the backstage team ? Looks like a lot of us are unable to make it work with heroku postgres databases.

Regards