OperationalError: SSL error: decryption failed or bad record mac

See original GitHub issue

Not sure if this is a django-rq issue or python-rq, so I figured I’d start here…

My application was working perfectly under Django 1.7.x.

I updated to Django 1.8.x and my workers blew up.

WARNING 11:29:35 worker 16516 140021590959936 Moving job to u'failed' queue
WARNING 2015-08-30 11:29:35,394 worker 16516 140021590959936 Moving job to u'failed' queue
ERROR 11:29:35 worker 16518 140021590959936 OperationalError: SSL error: decryption failed or bad record mac

Traceback (most recent call last):
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/rq/worker.py", line 568, in perform_job
    rv = job.perform()
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/rq/job.py", line 495, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/tank/code/uitintranet/intranet/tasks.py", line 131, in backup_router
    router = Router.objects.get(pk=router_pk)
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/aaron/.virtualenvs/uitintranet/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
OperationalError: SSL error: decryption failed or bad record mac

I checked, and updated rq from 0.5.4 to 0.5.5 and the error continues. If I run a command from my Django shell_plus without ‘.delay()’, it runs fine. If I run a command with ‘.delay()’ or I run a command using the scheduler, I get the error above.

I noticed a post on StackOverflow that basically says ‘close the DB connection at the beginning of each job’. (http://stackoverflow.com/questions/17523912/django-python-rq-databaseerror-ssl-error-decryption-failed-or-bad-record-mac)

I tested with one of my jobs, and it does fix the problem. But I’m worried about possible side effects, and having to add a few lines of code to hundreds of job definitions.

Issue Analytics

  • State:closed
  • Created 8 years ago
  • Comments:16

github_iconTop GitHub Comments

2reactions
darkpixelcommented, Sep 2, 2015

I totally misread celery/celery#634. It is definitely an SSL issue. But everything else django-related works fine.

manage.py runserver and manage.py shell_plus can interact with the DB without problems.

When I specifically force SSL to be disabled in Django (and it’s optional in Postgres)

      'cwdump': {
          'ENGINE': 'django.db.backends.postgresql_psycopg2',
          'NAME': 'uitintranet',
          'USER': 'uitintranet',
          'PASSWORD': '--redacted--',
          'HOST': '--redacted---',
         'OPTIONS': {
             'sslmode': 'disable',
         },

Everything works fine.

When I change ‘sslmode’ to ‘require’, Django operates without any problems and the connection is encrypted, but the workers start bombing.

A packet capture shows Postgres throwing a TCP reset and closing the connection.

1reaction
darkpixelcommented, Sep 4, 2015

Yes, adding --worker-class rq.SimpleWorker stops the database errors.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Fixing “SSL error: decryption failed or bad record mac” - Medium
The SSL error: decryption failed or bad record mac occurs either when the certificate is invalid or the message hash value has been...
Read more >
(psycopg2.OperationalError) SSL error: decryption failed or ...
A Fix for OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac ... The accepted solution suggests using ...
Read more >
decryption failed or bad record mac in multiprocessing
OperationalError : SSL error: decryption failed or bad record mac. The full code which makes the error import requests import multiprocessing ...
Read more >
decryption failed or bad record mac & SSL SYSCALL error ...
After a fresh nautobot start the error occurs. After some minutes it disappears... OperationalError at /dcim/devices/ SSL error: decryption ...
Read more >
SSL: decryption failed or bad record mac with upstream servers
In your case errors are reported at the "info" level as these are errors happening on a connection to a client. Such errors...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found