Helm deployment fails due to password authentication failed for user "backend-user"

See original GitHub issue

Issue

Helm deployment fails as the backend cannot access the DB Log of the backstage backend pod

(node:1) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./" in the "exports" field module resolution of the package at /app/node_modules/yam
Update this package.json to use a subpath pattern like "./*". 
(Use `node --trace-deprecation ...` to show where the warning was created)
2022-05-16T10:52:03.741Z backstage info Loaded config from app-config.yaml, app-config.development.yaml, env
2022-05-16T10:52:03.754Z backstage info 1 secrets found in the config which will be redacted
2022-05-16T10:52:03.773Z backstage info Created UrlReader predicateMux{readers=azure{host=dev.azure.com,authed=false},bitbucket{host=bitbucket.org,authed=false},github{host=github.com,authed=true},gitlab{host=gitlab.com,authed=false},awsS3{host=amazonaws.com,authed=false},fetch{}
Backend failed to start up, Error: Failed to connect to the database to make sure that 'backstage_plugin_catalog' exists, error: password authentication failed for user "backend-user"

and backstage-lighthouse error: password authentication failed for user "backend-user"

Log of the backstage-lighthouse pod

yarn run v1.22.4
$ node ./cjs/run.js
info: building express app... {"service":"lighthouse-audit-service","timestamp":"2022-05-16T10:53:07.570Z"}
error: password authentication failed for user "backend-user" {"name":"error","length":108,"severity":"FATAL","code":"28P01","file":"auth.c","line":"338","routin
e":"auth_failed","service":"lighthouse-audit-service","timestamp":"2022-05-16T10:53:32.486Z"}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Log of the postgresql DB reports also

2022-05-17 05:44:56.187 GMT [675] DETAIL:  Role "backend-user" does not exist.
    Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"
2022-05-17 05:45:08.962 GMT [690] FATAL:  password authentication failed for user "backend-user"

I get the same error if I try to log on using the password of the secret

kubectl -n backstage get secret/backstage-postgresql -o jsonpath='{.data.postgres-password}' | base64 -d
gmPqVycTqv%

kubectl exec -it backstage-postgresql-0 -n backstage -c postgresql -- bash -c "psql -U backend-user --password"
Password: gmPqVycTqv
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  password authentication failed for user "backend-user"
command terminated with exit code 2

Database and tables created

k exec -it backstage-postgresql-0 -n backstage -c postgresql -- bash -c "PGPASSWORD=gmPqVycTqv psql -U postgres -c \"\l+\""
                                                                    List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |  Size   | Tablespace |                Description
-----------+----------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 8529 kB | pg_default | default administrative connection database
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8377 kB | pg_default | unmodifiable empty database
           |          |          |             |             | postgres=CTc/postgres |         |            |
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8377 kB | pg_default | default template for new databases
           |          |          |             |             | postgres=CTc/postgres |         |            |
(3 rows)

and users 

postgres=# \du
                                   List of roles
 Role name |                         Attributes                         | Member of
-----------+------------------------------------------------------------+-----------
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

Remark: As you can see the user backend-user is not listed and this is why we cannot access the DB, tables, … and been authenticated

Steps to reproduce

kind cluster created using this bash script: https://github.com/snowdrop/k8s-infra/blob/main/kind/kind-reg-ingress.sh
npx @backstage/create-app
Name: backend
DB: postgresql
cd backend
yarn add --cwd packages/app graphql-ws
yarn build
yarn build-image -t backstage:dev

kind load docker-image backstage:dev

git clone https://github.com/backstage/backstage.git
cd contrib/chart/backstage
helm dependency update

DOMAIN_NAME="192.168.1.90.nip.io"
cat <<EOF > cfg.yml
backend:
  image:
    repository: backstage
    tag: dev
postgresql:
  service:
    port: 5432    
appConfig:
  app:
    baseUrl: https://backstage.$DOMAIN_NAME
    title: Backstage
  backend:
    baseUrl: https://backstage.$DOMAIN_NAME
    cors:
      origin: https://backstage.$DOMAIN_NAME
  lighthouse:
    baseUrl: https://backstage.$DOMAIN_NAME/lighthouse-api
  techdocs:
    storageUrl: https://backstage.$DOMAIN_NAME/api/techdocs/static/docs
    requestUrl: https://backstage.$DOMAIN_NAME/api/techdocs
EOF

kubectl create ns backstage
helm install -f cfg.yml -n backstage backstage .

helm uninstall backstage -n backstage

Issue Analytics

  • State:closed
  • Created a year ago
  • Reactions:1
  • Comments:11 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
cmoulliardcommented, May 18, 2022

I did a new test using as chart version for posgresql 9.8.12 and DB is well created, backend pod is able to access the DB

(node:1) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./" in the "exports" field module resolution of the package at /app/node_modules/yaml/pa
Update this package.json to use a subpath pattern like "./*".
(Use `node --trace-deprecation ...` to show where the warning was created)
2022-05-18T15:43:29.450Z backstage info Loaded config from app-config.yaml, app-config.development.yaml, env
2022-05-18T15:43:29.464Z backstage info 1 secrets found in the config which will be redacted
2022-05-18T15:43:29.479Z backstage info Created UrlReader predicateMux{readers=azure{host=dev.azure.com,authed=false},bitbucket{host=bitbucket.org,authed=false},
2022-05-18T15:43:29.808Z catalog info Performing database migration type=plugin
2022-05-18T15:43:30.419Z auth info Configuring "database" as KeyStore provider type=plugin
2022-05-18T15:43:30.497Z auth info Configuring provider, google type=plugin
2022-05-18T15:43:30.497Z auth warn Skipping google auth provider, Missing required config value at 'auth.providers.google.development.clientId' type=plugin
2022-05-18T15:43:30.498Z auth info Configuring provider, github type=plugin
2022-05-18T15:43:30.499Z auth warn Skipping github auth provider, Missing required config value at 'auth.providers.github.development.clientId' type=plugin
2022-05-18T15:43:30.499Z auth info Configuring provider, gitlab type=plugin
2022-05-18T15:43:30.500Z auth warn Skipping gitlab auth provider, Missing required config value at 'auth.providers.gitlab.development.clientId' type=plugin
2022-05-18T15:43:30.501Z auth info Configuring provider, okta type=plugin
2022-05-18T15:43:30.501Z auth warn Skipping okta auth provider, Missing required config value at 'auth.providers.okta.development.clientId' type=plugin
2022-05-18T15:43:30.502Z auth info Configuring provider, oauth2 type=plugin
2022-05-18T15:43:30.503Z auth warn Skipping oauth2 auth provider, Missing required config value at 'auth.providers.oauth2.development.clientId' type=plugin
2022-05-18T15:43:30.507Z techdocs info Creating Local publisher for TechDocs type=plugin
2022-05-18T15:43:30.510Z proxy info [HPM] Proxy created: /test  -> https://example.com type=plugin
2022-05-18T15:43:30.511Z proxy info [HPM] Proxy rewrite rule created: "^/api/proxy/test/?" ~> "/" type=plugin
2022-05-18T15:43:30.810Z search info Added DefaultCatalogCollatorFactory collator factory for type software-catalog type=plugin
2022-05-18T15:43:30.810Z search info Added DefaultTechDocsCollatorFactory collator factory for type techdocs type=plugin
2022-05-18T15:43:30.812Z search info Starting all scheduled search tasks. type=plugin
2022-05-18T15:43:30.818Z app info Serving static app content from /app/packages/app/dist type=plugin
2022-05-18T15:43:31.011Z app info Injecting env config into module-backstage.650db5d7.js type=plugin
2022-05-18T15:43:31.034Z search info Task worker starting: search_index_software_catalog, {"version":2,"cadence":"PT10M","initialDelayDuration":"PT3S","timeoutAf
2022-05-18T15:43:31.036Z search info Task worker starting: search_index_techdocs, {"version":2,"cadence":"PT10M","initialDelayDuration":"PT3S","timeoutAfterDurat
2022-05-18T15:43:31.141Z catalog warn No processor was able to handle reading of github:https://github.com/backstage/backstage/blob/master/plugins/scaffolder-bac
2022-05-18T15:43:31.145Z catalog warn No processor was able to handle reading of github:https://github.com/backstage/backstage/blob/master/packages/catalog-model
2022-05-18T15:43:31.179Z catalog warn No processor was able to handle reading of github:https://github.com/backstage/backstage/blob/master/plugins/techdocs-backe
2022-05-18T15:43:31.190Z catalog warn No processor was able to handle reading of github:https://github.com/backstage/backstage/blob/master/packages/catalog-model
2022-05-18T15:43:31.193Z catalog warn No processor was able to handle reading of github:https://github.com/backstage/backstage/blob/master/plugins/github-actions
2022-05-18T15:43:31.395Z app info Storing 0 updated assets and 279 new assets type=plugin
2022-05-18T15:43:32.104Z backstage info Initializing http server
2022-05-18T15:43:32.106Z backstage info Listening on :7007
2022-05-18T15:43:34.043Z search info Collating documents for software-catalog via DefaultCatalogCollatorFactory type=plugin
2022-05-18T15:43:34.050Z search info Collating documents for techdocs via DefaultTechDocsCollatorFactory type=plugin
2022-05-18T15:43:34.068Z backstage info ::ffff:127.0.0.1 - - [18/May/2022:15:43:34 +0000] "GET /api/catalog/entities?offset=0&limit=500 HTTP/1.1" 200 2 "-" "node
2022-05-18T15:43:34.069Z backstage info ::ffff:127.0.0.1 - - [18/May/2022:15:43:34 +0000] "GET /api/catalog/entities?filter=metadata.annotations.backstage.io%2Ft
2022-05-18T15:43:34.073Z search info Collating documents for software-catalog succeeded type=plugin
2022-05-18T15:43:34.077Z search info Collating documents for techdocs succeeded type=plugin

If I also deploy the frontend, that works 😃

Screenshot 2022-05-18 at 18 29 44

1reaction
cmoulliardcommented, May 18, 2022

Here is what I do to install the front and the backend

DOMAIN_NAME="192.168.1.90.nip.io"
cat <<EOF > cfg.yml
frontend:
  enabled: true
backend:
  image:
    repository: backstage
    tag: dev
postgresql:
  service:
    port: 5432    
appConfig:
  app:
    baseUrl: https://backstage.$DOMAIN_NAME
    title: Backstage
  backend:
    baseUrl: https://backstage.$DOMAIN_NAME
    cors:
      origin: https://backstage.$DOMAIN_NAME
  lighthouse:
    baseUrl: https://backstage.$DOMAIN_NAME/lighthouse-api
  techdocs:
    storageUrl: https://backstage.$DOMAIN_NAME/api/techdocs/static/docs
    requestUrl: https://backstage.$DOMAIN_NAME/api/techdocs
EOF

kubectl create ns backstage
helm install -f cfg.yml -n backstage backstage .

Note: I created a ticket to improve the documentation to build the frontend more easily - see #11598

Read more comments on GitHub >

github_iconTop Results From Across the Web

[postgresql] bug: password authentication failed for user ...
Admin user (postgres) password fails with or without default settings. Steps to reproduce the issue: Setup a values.yaml file like below:.
Read more >
password authentication failed for user "postgres" when using ...
I am able to run helm, postgresql chart ...
Read more >
Using awx-operator 1.0.0 by helm chart - Google Groups
Using awx-operator 1.0.0 by helm chart - password authentication failed for user "awx". 0 views. Skip to first unread message.
Read more >
Troubleshoot VMware Application Catalog Helm Charts
VMware Application Catalog Helm charts provide an easy way to ... -p 5432 psql: FATAL: password authentication failed for user "postgres".
Read more >
Can't log in (helm) or reset password (#2005) · Issues - GitLab
Either authentication, inability to connect, or even just getting connections dropped. When you encounter these things, be sure to check the ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found