redis.exceptions.TimeoutError: Timeout reading from socket

Hi Suor,

Firstly thanks for this amazing library. Everyone at my company loves it. So, we have been using this paired with Django REST framework for a large datastore(Postgres db). This datastore gets new data written to it every 20-30 minutes using bulk_create. We noticed that there happens a socket timeout to redis. It doesn’t happen often and maybe once in several hours and crashes our db writes. I understand that cacheops invalidates on bulk_creates and for some reason the socket connection times out. I tried increases the timeout using cacheops to 10 seconds (a lot, I know) from 4 seconds and still doesn’t help.

Error:

Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 181, in _read_from_socket
   data = recv(self._sock, socket_read_size)
 File "/usr/local/lib/python3.7/site-packages/redis/_compat.py", line 71, in recv
   return sock.recv(*args, **kwargs)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "article.py", line 125, in <module>
   add_data(cursor)
 File "article.py", line 107, in add_data
   entity = models.Entity.objects.filter(dow_jones_fcode = company.upper()).first()
 File "/usr/local/lib/python3.7/site-packages/cacheops/query.py", line 360, in first
   return self._no_monkey.first(self._clone().cache())
 File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 605, in first
   for obj in (self if self.ordered else self.order_by('pk'))[:1]:
 File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 268, in __iter__
   self._fetch_all()
 File "/usr/local/lib/python3.7/site-packages/cacheops/query.py", line 295, in _fetch_all
   self._cache_results(cache_key, self._result_cache)
 File "/usr/local/lib/python3.7/site-packages/cacheops/query.py", line 181, in _cache_results
   self._cond_dnfs, self._cacheprofile['timeout'], dbs=[self.db])
 File "/usr/local/lib/python3.7/site-packages/cacheops/query.py", line 47, in cache_thing
   timeout
 File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 3575, in __call__
   return client.evalsha(self.sha, len(keys), *args)
 File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 2761, in evalsha
   return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
 File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 775, in execute_command
   return self.parse_response(connection, command_name, **options)
 File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 789, in parse_response
   response = connection.read_response()
 File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 636, in read_response
   raise e
 File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 633, in read_response
   response = self._parser.read_response()
 File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 291, in read_response
   response = self._buffer.readline()
 File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 223, in readline
   self._read_from_socket()
 File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 194, in _read_from_socket
   raise TimeoutError("Timeout reading from socket")
redis.exceptions.TimeoutError: Timeout reading from socket

settings.py

# caching
CACHEOPS_REDIS = {
    'host': os.environ['DJANGO_CACHE_HOST'],  # redis-server is on same machine
    'port': os.environ['DJANGO_CACHE_PORT'],  # default redis port
    'db': os.environ['DJANGO_CACHE_DB_NUMBER'],  # SELECT non-default redis database
    'socket_timeout': int(os.environ['DJANGO_CACHE_SOCKET_TIMEOUT']),  # connection timeout in seconds
    'password': os.environ['DJANGO_CACHE_PASSWORD'],
}

CACHEOPS_DEFAULTS = {
    # 60 * 60 * 24 = 1 day
    'timeout': 60 * 60 * 24 * int(os.environ['DJANGO_CACHE_DEFAULT_EXPIRY_DAYS']),
    'ops': {'get', 'fetch', 'count', 'aggregate'},
    'local_get': True,
    'cache_on_save': True,
}

# uses defaults from above
CACHEOPS = {
    'auth.user': {'ops': 'get', 'timeout': 60 * 60 * 24},
    'auth.permission': {'ops': 'all', 'timeout': 60 * 60 * 24},
    'tang_app.DailyAdjClose': {},
    'tang_app.Industry': {},
    'tang_app.Country': {},
    'tang_app.Entity': {},
    'tang_app.Article': {},
    'tang_app.Source': {},
    'tang_app.Publisher': {},
}

Any advice is appreciated. Thanks again for your work.