aziot-edged fails to start and load settings

Expected Behavior

After running “sudo iotedge system status”, I should get the following results: System services: aziot-edged Ready aziot-identityd Ready aziot-keyd Ready aziot-certd Ready aziot-tpmd Ready And after running “sudo iotedge list”, I should see the EdgeAgent and EdgeHub running.

Current Behavior

When I ran “sudo iotedge list”, I got the following error messages: A module runtime error occurred caused by: A module runtime error occurred caused by: connection error: Connection reset by peer (os error 104) caused by: Connection reset by peer (os error 104) Then I ran “sudo iotedge system status”, and got the following error messages: System services: aziot-edged Down - failed aziot-identityd Ready aziot-keyd Ready aziot-certd Ready aziot-tpmd Ready

aziot-edged is in a bad state because:

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

Run sudo iotedge config mp --connection-string ‘My primary connection string’. It looks IoT Edge was configured successfully and I got the following messages:

Azure IoT Edge has been configured successfully! The configuration has been written to /etc/aziot/config.toml To apply the new configuration to services, run: sudo iotedge config apply -c ‘/etc/aziot/config.toml’
Run sudo iotedge config apply -c ‘/etc/aziot/config.toml’ and I got the following messages:

Note: Symmetric key will be written to /var/secrets/aziot/keyd/device-id Azure IoT Edge has been configured successfully!

Restarting service for configuration to take effect… Stopping aziot-edged.service…Stopped! Stopping aziot-identityd.service…Stopped! Stopping aziot-keyd.service…Stopped! Stopping aziot-certd.service…Stopped! Stopping aziot-tpmd.service…Stopped! Starting aziot-edged.mgmt.socket…Started! Starting aziot-edged.workload.socket…Started! Starting aziot-identityd.socket…Started! Starting aziot-keyd.socket…Started! Starting aziot-certd.socket…Started! Starting aziot-tpmd.socket…Started! Starting aziot-edged.service…Started! Done.
Run “sudo iotedge list” and I got the following errors: A module runtime error occurred caused by: A module runtime error occurred caused by: connection error: Connection reset by peer (os error 104) caused by: Connection reset by peer (os error 104)

Context (Environment)

Output of `iotedge check`

Click here


Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
× read all preloaded certificates from the Certificates Service - Error
    could not load cert with ID "aziot-edged-trust-bundle"

    Caused by:
        parameter "id" has an invalid value
        caused by: not found
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
× aziot-edged configuration is well-formed - Error
    Error running check: The IoT Edge daemon's configuration file is not well-formed.
√ configuration up-to-date with config.toml - OK
‼ aziot-edge package is up-to-date - Warning
    Installed IoT Edge daemon has version 1.4.4 but 1.4.3 is the latest stable version available.
    Please see https://aka.ms/iotedge-update-runtime for update instructions.
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.

Connectivity checks
-------------------
19 check(s) succeeded.
3 check(s) raised warnings. Re-run with --verbose for more details.
2 check(s) raised errors. Re-run with --verbose for more details.
14 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

Host OS [e.g. Ubuntu 18.04, Windows Server IoT 2019]: Dunfell 5.7.1
Architecture [e.g. amd64, arm32, arm64]: arm64
Container OS [e.g. Linux containers, Windows containers]: Linux containers

Runtime Versions

aziot-edged [run iotedge version]: 1.4.4
Edge Agent [image tag (e.g. 1.0.0)]: 1.4
Edge Hub [image tag (e.g. 1.0.0)]: 1.4
Docker/Moby [run docker version]: 19.03.14

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

aziot-edged logs


System services:
    aziot-edged             Down - failed
    aziot-identityd         Running
    aziot-keyd              Running
    aziot-certd             Running
    aziot-tpmd              Ready

aziot-edged is in a bad state because:
aziot-edged.service: Down - failed : Printing the last 10 log lines.
-- Logs begin at Tue 2022-12-20 15:03:38 UTC, end at Tue 2022-12-20 18:39:30 UTC. --
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: aziot-edged.service: Failed with result 'exit-code'.
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: Started Azure IoT Edge daemon.
Dec 20 18:05:00 colibri-imx8x-06995462 aziot-edged[4867]: 2022-12-20T18:05:00Z [INFO] - Starting Azure IoT Edge Daemon
Dec 20 18:05:00 colibri-imx8x-06995462 aziot-edged[4867]: 2022-12-20T18:05:00Z [INFO] - Version - 1.4.4
Dec 20 18:05:00 colibri-imx8x-06995462 aziot-edged[4867]: 2022-12-20T18:05:00Z [ERR!] - Failed to load settings: could not read config from /etc/aziot/edged/config.d/00-super.toml
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: aziot-edged.service: Main process exited, code=exited, status=153/n/a
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: aziot-edged.service: Failed with result 'exit-code'.
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: aziot-edged.service: Start request repeated too quickly.
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: aziot-edged.service: Failed with result 'exit-code'.
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: Failed to start Azure IoT Edge daemon.

aziot-edged.mgmt.socket: Down - failed : Printing the last 10 log lines.
-- Logs begin at Tue 2022-12-20 15:03:38 UTC, end at Tue 2022-12-20 18:39:30 UTC. --
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Closed Azure IoT Edge daemon management socket.
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Stopping Azure IoT Edge daemon management socket.
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Starting Azure IoT Edge daemon management socket.
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Listening on Azure IoT Edge daemon management socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: aziot-edged.mgmt.socket: Succeeded.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Closed Azure IoT Edge daemon management socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Stopping Azure IoT Edge daemon management socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Starting Azure IoT Edge daemon management socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Listening on Azure IoT Edge daemon management socket.
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: aziot-edged.mgmt.socket: Failed with result 'service-start-limit-hit'.

aziot-edged.workload.socket: Down - failed : Printing the last 10 log lines.
-- Logs begin at Tue 2022-12-20 15:03:38 UTC, end at Tue 2022-12-20 18:39:30 UTC. --
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Closed Azure IoT Edge daemon workload socket.
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Stopping Azure IoT Edge daemon workload socket.
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Starting Azure IoT Edge daemon workload socket.
Dec 20 16:17:50 colibri-imx8x-06995462 systemd[1]: Listening on Azure IoT Edge daemon workload socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: aziot-edged.workload.socket: Succeeded.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Closed Azure IoT Edge daemon workload socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Stopping Azure IoT Edge daemon workload socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Starting Azure IoT Edge daemon workload socket.
Dec 20 18:03:17 colibri-imx8x-06995462 systemd[1]: Listening on Azure IoT Edge daemon workload socket.
Dec 20 18:05:00 colibri-imx8x-06995462 systemd[1]: aziot-edged.workload.socket: Failed with result 'service-start-limit-hit'.

-- Logs begin at Tue 2022-12-20 15:03:38 UTC. --
Dec 20 18:05:19 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:05:19Z [ERR!] - !!! caused by: could not load key pair
Dec 20 18:05:19 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:05:19Z [ERR!] - !!! caused by: could not load key pair: AZIOT_KEYS_RC_ERR_EXTERNAL
Dec 20 18:30:48 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:30:48Z [ERR!] - error:0909006C:PEM routines:get_name:no start line:../openssl-1.1.1s/crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY
Dec 20 18:30:48 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:30:48Z [ERR!] - !!! internal error
Dec 20 18:30:48 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:30:48Z [ERR!] - !!! caused by: could not load key pair
Dec 20 18:30:48 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:30:48Z [ERR!] - !!! caused by: could not load key pair: AZIOT_KEYS_RC_ERR_EXTERNAL
Dec 20 18:38:32 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:38:32Z [ERR!] - error:0909006C:PEM routines:get_name:no start line:../openssl-1.1.1s/crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY
Dec 20 18:38:32 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:38:32Z [ERR!] - !!! internal error
Dec 20 18:38:32 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:38:32Z [ERR!] - !!! caused by: could not load key pair
Dec 20 18:38:32 colibri-imx8x-06995462 aziot-keyd[3299]: 2022-12-20T18:38:32Z [ERR!] - !!! caused by: could not load key pair: AZIOT_KEYS_RC_ERR_EXTERNAL

edge-agent logs


<Paste here between the triple backticks>

edge-hub logs


<Paste here between the triple backticks>

Additional Information

After completing a yocto build containing meta-iotedge layer (dunfell) , it looks iotedge 1.4.4 was succesfully build into the OS. Then I was trying to connect the IoT Edge device to the IoT hub using connection string. However, after configuring the Azure IoT Edge, I can’t get aziot-edged running. The log files indicated aziot-edged always failed to load settings (ERROR: No such file or directory (os error 2)), but I found 00-super.toml file in /etc/aziot/edged/config.d/ folder. Please see the following log files for more information. I’m not sure if I configured the IoT Edge in a correct way, so could you please help take a look at the following logs and see what may go wrong?

Issue Analytics

State:
Created 9 months ago
Comments:10 (4 by maintainers)

Top GitHub Comments

1reaction

onalante-msftcommented, Dec 24, 2022

The configuration is well-formed (structurally). I wonder if the issue is something in the Yocto environment itself. It seems that loading settings fails with ERRNO 2 when the homedir directory does not exist: ref. Does /var/lib/aziot/edged exist?

1reaction

onalante-msftcommented, Dec 21, 2022

Two things:

0700 for directories and 0600 should be fine since the iotedge is the owning user. You can revert to the original permissions.
I will check the structure of 00-super.toml since the iotedge check command is complaining about a malformed configuration.