Minion did not return. [No response] - system.reboot

See original GitHub issue

Description of Issue

i am trying to reboot linux host during orchestration ( deployment process). i have tried multiple ways, as soon as it hit reboot.sls i got error message.

Setup

reboot.sls

reboot_minions:
  salt.function:
    - name: system.reboot
    - at_time: 1
    - tgt: {{pillar['match']|join(',')}}
    - tgt_type: list

wait_for_reboots:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list: {{pillar['match']}}
    - require:
      - salt: reboot_minions

also tried with code below and run via orchestration. reboot.sls

final_housekeeping:
   module.run:
      - name: system.reboot
      - wait_for_reboot: True

init.sls

run minion restart 1:
  salt.state:
    - tgt: {{ target_minion }}
    - sls:
      - {{ slspath }}.reboot

wait for minion to restart 1:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list:
      - {{ target_minion }}
    - require:
      - salt: run minion restart 1

Steps to Reproduce Issue

[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] LazyLoaded state.orchestrate
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/run/20201230204901837092/new; data = {'fun': 'runner.state.orchestrate', 'jid': '20201230204901837092', 'user': 'root', 'fun_args': ['sqlprovision', {'pillar': {'target': 'wh-1c385b6305', 'login': 'xxxx', 'pass': 'xxxxx', 'sapass': 'xxxxxx', 'inputfile': '/home/SQLInstall/linux_gss_inputfile.txt'}, 'orchestration_jid': '20201230204901837092'}], '_stamp': '2020-12-30T20:49:04.312062'}
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.sls
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] key: test, ret: _|-
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Determining pillar cache
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: /srv/pillar/top.sls
[DEBUG   ] Jinja search path: ['/srv/pillar', '/srv/spm/pillar']
[PROFILE ] Time (in seconds) to render '/srv/pillar/top.sls' using 'jinja' renderer: 0.011786699295043945
[DEBUG   ] Rendered data from file: /srv/pillar/top.sls:
base:
  '*':
    - InputFilePath

[DEBUG   ] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['InputFilePath'])]))])
[PROFILE ] Time (in seconds) to render '/srv/pillar/top.sls' using 'yaml' renderer: 0.00030517578125
[DEBUG   ] LazyLoaded confirm_top.confirm_top
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] compound_match: wh-140a786872.svr.us.jpmchase.net_master ? *
[DEBUG   ] LazyLoaded glob_match.match
[DEBUG   ] compound_match wh-140a786872.svr.us.jpmchase.net_master ? "*" => "True"
[DEBUG   ] compile template: /srv/pillar/InputFilePath.sls
[DEBUG   ] Jinja search path: ['/srv/pillar', '/srv/spm/pillar']
[PROFILE ] Time (in seconds) to render '/srv/pillar/InputFilePath.sls' using 'jinja' renderer: 0.0006053447723388672
[DEBUG   ] Rendered data from file: /srv/pillar/InputFilePath.sls:
# Set path to input file to be stored on Pillar

inputfile: "/home/SQLInstall/linux_gss_inputfile.txt"

[DEBUG   ] Results of YAML rendering:
OrderedDict([('inputfile', '/home/SQLInstall/linux_gss_inputfile.txt')])
[PROFILE ] Time (in seconds) to render '/srv/pillar/InputFilePath.sls' using 'yaml' renderer: 0.00016188621520996094
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Could not find file 'salt://sqlprovision.sls' in saltenv 'base'
[DEBUG   ] In saltenv 'base', looking at rel_path 'sqlprovision/init.sls' to resolve 'salt://sqlprovision/init.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/master/files/base/sqlprovision/init.sls' to resolve 'salt://sqlprovision/init.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://sqlprovision/init.sls'
[DEBUG   ] No dest file found
[INFO    ] Fetching file from saltenv 'base', ** done ** 'sqlprovision/init.sls'
[DEBUG   ] compile template: /var/cache/salt/master/files/base/sqlprovision/init.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/master/files/base']
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] LazyLoaded pillar.get
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/files/base/sqlprovision/init.sls' using 'jinja' renderer: 0.024741411209106445
[DEBUG   ] Rendered data from file: /var/cache/salt/master/files/base/sqlprovision/init.sls:







run minion restart 1:
  salt.state:
    - tgt: wh-1c385b6305
    - sls:
      - sqlprovision.minion_reboot

wait for minion to restart 1:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list:
      - wh-1c385b6305
    - require:
      - salt: run minion restart 1

[DEBUG   ] Results of YAML rendering:
OrderedDict([('run minion restart 1', OrderedDict([('salt.state', [OrderedDict([('tgt', 'wh-1c385b6305')]), OrderedDict([('sls', ['sqlprovision.minion_reboot'])])])])), ('wait for minion to restart 1', OrderedDict([('salt.wait_for_event', [OrderedDict([('name', 'salt/minion/*/start')]), OrderedDict([('id_list', ['wh-1c385b6305'])]), OrderedDict([('require', [OrderedDict([('salt', 'run minion restart 1')])])])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/files/base/sqlprovision/init.sls' using 'yaml' renderer: 0.0004239082336425781
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] LazyLoaded salt.state
[INFO    ] Running state [run minion restart 1] at time 20:49:04.922186
[INFO    ] Executing state salt.state for [run minion restart 1]
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/run/20201230204901837092/args; data = {'type': 'state', 'tgt': 'wh-1c385b6305', 'name': 'run minion restart 1', 'args': {'arg': ['sqlprovision.minion_reboot'], 'kwarg': {'queue': False, 'concurrent': False}, 'ret': '', 'timeout': None, 'tgt_type': 'glob', 'ssh': False, 'expect_minions': True}, '_stamp': '2020-12-30T20:49:04.924076'}
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] LazyLoaded saltutil.cmd
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'wh-140a786872.svr.us.jpmchase.net_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/39/691a1a433f982acb9f45475ee8a99d599155dae2f2fe816e871f253edd9257/.minions.p
[DEBUG   ] get_iter_returns for jid 20201230204904974268 sent to {'wh-1c385b6305'} will timeout at 20:49:09.978757
[DEBUG   ] Checking whether jid 20201230204904974268 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'wh-140a786872.svr.us.jpmchase.net_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] retcode missing from client return
[DEBUG   ] Closing IPCMessageSubscriber instance
[WARNING ] Output from salt state not highstate
[ERROR   ] {'out': 'highstate', 'ret': {'wh-1c385b6305': False}}
[INFO    ] Completed state [run minion restart 1] at time 20:49:20.059995 (duration_in_ms=15137.808)
[DEBUG   ] File /var/cache/salt/master/accumulator/140440397671224 does not exist, no need to cleanup
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] Sending event: tag = salt/run/20201230204901837092/ret; data = {'fun': 'runner.state.orchestrate', 'jid': '20201230204901837092', 'user': 'root', 'fun_args': ['sqlprovision', {'pillar': {'target': 'wh-1c385b6305', 'login': ' xxxxx', 'pass': 'xxxx', 'sapass': 'xxxx', 'inputfile': '/home/SQLInstall/linux_gss_inputfile.txt'}, 'orchestration_jid': '20201230204901837092'}], '_stamp': '2020-12-30T20:49:20.063392', 'return': {'data': {'wh-140a786872.svr.us.jpmchase.net_master': {'salt_|-run minion restart 1_|-run minion restart 1_|-state': {'name': 'run minion restart 1', 'changes': {'out': 'highstate', 'ret': {'wh-1c385b6305': False}}, 'comment': 'Run failed on minions: wh-1c385b6305', 'result': False, '__sls__': 'sqlprovision', '__run_num__': 0, 'start_time': '20:49:04.922187', 'duration': 15137.808, '__id__': 'run minion restart 1'}, 'salt_|-wait for minion to restart 1_|-salt/minion/*/start_|-wait_for_event': {'changes': {}, 'result': False, 'duration': 0.004, 'start_time': '20:49:20.060410', 'comment': 'One or more requisite failed: sqlprovision.run minion restart 1', '__run_num__': 1, '__sls__': 'sqlprovision'}}}, 'outputter': 'highstate', 'retcode': 1}, 'success': False}
[DEBUG   ] LazyLoaded highstate.output
wh-140a786872.svr.us.jpmchase.net_master:
----------
          ID: run minion restart 1
    Function: salt.state
      Result: False
     Comment: Run failed on minions: wh-1c385b6305
     Started: 20:49:04.922187
    Duration: 15137.808 ms
     Changes:
              wh-1c385b6305:
                  False
----------
          ID: wait for minion to restart 1
    Function: salt.wait_for_event
        Name: salt/minion/*/start
      Result: False
     Comment: One or more requisite failed: sqlprovision.run minion restart 1
     Started: 20:49:20.060410
    Duration: 0.004 ms
     Changes:

Summary for wh-140a786872.svr.us.jpmchase.net_master
------------
Succeeded: 0 (changed=1)
Failed:    2
------------
Total states run:     2
Total run time:  15.138 s

Versions Report

Salt Version:
           Salt: 2019.2.7

Dependency Versions:
           cffi: 1.9.1
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.11.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug 13 2020, 07:46:32)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.7 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.18.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.7 Maipo

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
robertpenberthycommented, Dec 31, 2020

We’ve been using this to reboot our Hosts and has been working for us for a while now. It’s slightly different than what you have, but maybe you can tweak yours to work. The only real difference I see is you have at_time: 1 defined in your system.reboot statement and I’m passing the 1 as a positional argument, with the obvious difference also being how we determine for which hosts to reboot and wait.

{% set deployment = salt['pillar.get']('deployment', none) %}
{% set tgt = 'G@deployment:{0}'.format(deployment)  %}
{% set hosts = salt['saltutil.runner']('mine.get', tgt=tgt, fun='ip', tgt_type='compound') %}

{{ sls }}-hosts-restart:
  salt.function:
    - name: system.reboot
    - arg: [1]
    - tgt: {{ tgt }}
    - tgt_type: compound

{{ sls }}-hosts-wait-for-reboot:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list: {{ hosts.keys()|list|yaml }}
    - timeout: 600
    - require:
      - salt: {{ sls }}-hosts-restart
1reaction
timmotejcommented, Jun 4, 2021

We’ve been using this to reboot our Hosts and has been working for us for a while now. It’s slightly different than what you have, but maybe you can tweak yours to work. The only real difference I see is you have at_time: 1 defined in your system.reboot statement and I’m passing the 1 as a positional argument, with the obvious difference also being how we determine for which hosts to reboot and wait.

{% set deployment = salt['pillar.get']('deployment', none) %}
{% set tgt = 'G@deployment:{0}'.format(deployment)  %}
{% set hosts = salt['saltutil.runner']('mine.get', tgt=tgt, fun='ip', tgt_type='compound') %}

{{ sls }}-hosts-restart:
  salt.function:
    - name: system.reboot
    - arg: [1]
    - tgt: {{ tgt }}
    - tgt_type: compound

{{ sls }}-hosts-wait-for-reboot:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list: {{ hosts.keys()|list|yaml }}
    - timeout: 600
    - require:
      - salt: {{ sls }}-hosts-restart

The argument - arg: [1] was that missing piece in the puzzle: my servers were switching off the salt service too fast and it didn’t return the response. With that line everything works great! Thanks @robertpenberthy !

Read more comments on GitHub >

github_iconTop Results From Across the Web

Salt minion returns no response after being accepted
Minion did not return. [No response]. Makes me think the salt-minion process is not running. (those other two errors are expected behavior ...
Read more >
Minion did not return. [No response] appears occasionally ...
[No response]". If i rebooted the salt-minion, next almost 24 test.ping worked and then this problem occurred again.
Read more >
Minion did not return. [No response] - RobViT
Check if your minion is running. On Linux: · Check if you can resolve the salt master. If not, add to dns or...
Read more >
Salt: Minion did not return but salt-call is working - Server Fault
The problem isn't that the salt client (run on the master) is not waiting long enough, it's that the response the minion returns...
Read more >
Troubleshooting the Salt Minion - Salt Project Documentation
Salt Minion Doesn't Return Anything While Running Jobs Locally ... When a command being run via Salt takes a very long time to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found