Exception not caught "Socket exception: Connection reset by peer (104)"

See original GitHub issue

So been wresting with this one for a while. Just saw the note in the repo about opening a ticket for exceptions not caught, so figured this would be the best place to start. My application is fairly large, but the issue is identifiable.

In the app I’m writing, it’s for doing network device upgrades. Regardless of the device, sometimes you’ll get a prompt before it reboots, sometimes you wont. In the cases where you don’t get one and the napalm waits out the netmiko delay_factor, we get the following:

Error trying to Napalm disconnect: ‘Search pattern never detected in send_command_expect: hostname#’

The full function is this:

def device_disconnect_napalm(ssh_napalm=None, log=None):
    try:
        if ssh_napalm:
            if napalm_is_connected(ssh_napalm=ssh_napalm):
                time.sleep(2)
                ssh_napalm.close()
            ssh_napalm = False
            log.info('Successfully disconnected the Napalm connection.')
    except Exception as e:
        log.warning("Error trying to Napalm disconnect: " + repr(str(e)) + "  This is normally OK on a disconnect during reboot, so just an FYI.")
        log.warning("If a socket exception occurs, also safe to ignore.  For some reason I can't catch it.")
        ssh_napalm = False
        pass
    return (ssh_napalm)

You can see my frustration in the log exception (lol)

So, the socket error is (probably) coming from paramiko transport. That kind of makes sense, seeing that none of the 3 modules below the hood (napalm > netmiko > paramiko) know that the device kicked us yet. It’s normally just this: Socket exception: Connection reset by peer (104)

I can’t catch it though. It’s really odd, it just prints it to the screen (which is annoying when the logger is going, for obvious reasons). Is this something we just need to add to the base exceptions file? That’s literally the only log I get 😦

Issue Analytics

State:
Created 4 years ago
Comments:8 (5 by maintainers)

Top GitHub Comments

1reaction

sanseihappacommented, Jun 7, 2019

Somewhat anecdotal observation: paramiko 2.4.2 with Python 2.7.x, and didn’t observe this issue.

1reaction

sanseihappacommented, Jun 7, 2019

Seeing the same. I’m observing two different failure modes, both during _check_banner: EOFError and ConnectionResetError.

paramiko 2.4.2 on Python 3.7:

05-Jun-2019 11:24:49	  File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 2138, in _check_banner
05-Jun-2019 11:24:49	    buf = self.packetizer.readline(timeout)
05-Jun-2019 11:24:49	  File "/usr/local/lib/python3.7/site-packages/paramiko/packet.py", line 367, in readline
05-Jun-2019 11:24:49	    buf += self._read_timeout(timeout)
05-Jun-2019 11:24:49	  File "/usr/local/lib/python3.7/site-packages/paramiko/packet.py", line 563, in _read_timeout
05-Jun-2019 11:24:49	    raise EOFError()
05-Jun-2019 11:24:49	EOFError
05-Jun-2019 11:24:49
05-Jun-2019 11:24:49	During handling of the above exception, another exception occurred:
05-Jun-2019 11:24:49
05-Jun-2019 11:24:49	Traceback (most recent call last):
05-Jun-2019 11:24:49	  File "/usr/local/lib/python3.7/site-packages/paramiko/client.py", line 397, in connect
05-Jun-2019 11:24:49	    t.start_client(timeout=timeout)
05-Jun-2019 11:24:49	  File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 587, in start_client
05-Jun-2019 11:24:49	    raise e
05-Jun-2019 11:24:49	  File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 1966, in run
05-Jun-2019 11:24:49	    self._check_banner()
05-Jun-2019 11:24:49	  File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 2143, in _check_banner
05-Jun-2019 11:24:49	    "Error reading SSH protocol banner" + str(e)
05-Jun-2019 11:24:49	paramiko.ssh_exception.SSHException: Error reading SSH protocol banner```

```2019-06-05 16:02:00,791 DEBUG:paramiko.transport:starting thread (client mode): 0xb4243908
2019-06-05 16:02:00,791 DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.4.2
2019-06-05 16:02:00,794 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner[Errno 104] Connection reset by peer
2019-06-05 16:02:00,795 ERROR:paramiko.transport:Traceback (most recent call last):
2019-06-05 16:02:00,795 ERROR:paramiko.transport:  File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 2138, in _check_banner
2019-06-05 16:02:00,795 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2019-06-05 16:02:00,795 ERROR:paramiko.transport:  File "/usr/local/lib/python3.7/site-packages/paramiko/packet.py", line 367, in readline
2019-06-05 16:02:00,795 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2019-06-05 16:02:00,795 ERROR:paramiko.transport:  File "/usr/local/lib/python3.7/site-packages/paramiko/packet.py", line 561, in _read_timeout
2019-06-05 16:02:00,795 ERROR:paramiko.transport:    x = self.__socket.recv(128)
2019-06-05 16:02:00,795 ERROR:paramiko.transport:ConnectionResetError: [Errno 104] Connection reset by peer
2019-06-05 16:02:00,795 ERROR:paramiko.transport:
2019-06-05 16:02:00,795 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2019-06-05 16:02:00,795 ERROR:paramiko.transport:
2019-06-05 16:02:00,795 ERROR:paramiko.transport:Traceback (most recent call last):
2019-06-05 16:02:00,795 ERROR:paramiko.transport:  File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 1966, in run
2019-06-05 16:02:00,795 ERROR:paramiko.transport:    self._check_banner()
2019-06-05 16:02:00,795 ERROR:paramiko.transport:  File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 2143, in _check_banner
2019-06-05 16:02:00,795 ERROR:paramiko.transport:    "Error reading SSH protocol banner" + str(e)
2019-06-05 16:02:00,795 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer
2019-06-05 16:02:00,795 ERROR:paramiko.transport:```