Operation failed - received 0 responses and 1 failures: READ_TOO_MANY_TOMBSTONES when using where=

The last couple of weeks I’m doing some small performance test on Stargate for the Document API. I’ve a small docker setup with 3 cassandra nodes and 1 Stargate node. I bulk load 850.000 documents with the PUT method and trying to get data with the WHERE filter to test the performance. In version 1.0.9 I’m getting the READ_TOO_MANY_TOMBSTONES error on some WHERE filter requests, not on all. For instance when I use in Postman the WHERE filter on zipcode:

{{baseUrl}}/v2/namespaces/test/collections/dossiers?page-size=10&where={ "zipcode": { "$eq": "6521MH" } }

… the request takes 1 minute and 23.23 seconds. A request using the WHERE filter on house number:

{{baseUrl}}/v2/namespaces/test/collections/dossiers?page-size=10&where={"houseNumber": {"$eq": "21" } }

… takes 189ms. And using the WHERE filter on place:

{{baseUrl}}/v2/namespaces/test/collections/dossiers?page-size=10&where={"place": {"$eq": "Putten" } }

… the request failed with an HTTP 500 Internal Server error. The response in Postman was “org.apache.cassandra.stargate.exceptions.ReadFailureException: Operation failed - received 0 responses and 1 failures: READ_TOO_MANY_TOMBSTONES from 172.21.0.2: 7000”.

ERROR [2021-02-23 18:20:17,184] io.stargate.web.docsapi.resources.DocumentResourceV2: Error when executing request
! org.apache.cassandra.stargate.exceptions.ReadFailureException: Operation failed - received 0 responses and 1 failures: READ_TOO_MANY_TOMBSTONES from 172.21.0.2:7000
! at io.stargate.db.cassandra.impl.Conversion.toExternal(Conversion.java:315)
! at io.stargate.db.cassandra.impl.Conversion.convertInternalException(Conversion.java:472)
! at io.stargate.db.cassandra.impl.CassandraPersistence$CassandraConnection.lambda$executeRequestOnExecutor$0(CassandraPersistence.java:443)
! at io.stargate.db.cassandra.impl.CassandraPersistence.lambda$runOnExecutor$1(CassandraPersistence.java:268)
! ... 5 common frames omitted
! Suppressed: org.apache.cassandra.exceptions.ReadFailureException: Operation failed - received 0 responses and 1 failures
!       at org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:137)
!       at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:143)
!       at org.apache.cassandra.service.StorageProxy$SingleRangeResponse.waitForResponse(StorageProxy.java:2071)
!       at org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:2081)
!       at org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:2054)
!       at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
!       at org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:144)
!       at org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:111)
!       at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:105)
!       at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2129)
!       at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2092)
!       at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
!       at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92)
!       at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:786)
!       at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:438)
!       at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:416)
!       at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:289)
!       at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117)
!       at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225)
!       at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
!       at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
!       at io.stargate.db.cassandra.impl.StargateQueryHandler.processPrepared(StargateQueryHandler.java:199)
!       at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
!       at io.stargate.db.cassandra.impl.CassandraPersistence$CassandraConnection.lambda$executeRequestOnExecutor$0(CassandraPersistence.java:436)
!       ... 6 common frames omitted
! Causing: java.util.concurrent.CompletionException: org.apache.cassandra.stargate.exceptions.ReadFailureException: Operation failed - received 0 responses and 1 failures: READ_TOO_MANY_TOMBSTONES from 172.21.0.2:7000
! at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
! at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
! at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661)
! at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
! at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
! at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
! at io.stargate.db.cassandra.impl.CassandraPersistence.lambda$runOnExecutor$1(CassandraPersistence.java:277)
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
! at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
! at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
! at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113)
! at java.lang.Thread.run(Thread.java:748)
172.21.0.1 - - [23/Feb/2021:18:20:17 +0000] "GET /v2/namespaces/test/collections/dossiers?page-size=10&where={%22place%22:%20{%22$eq%22:%20%22Putten%22%20}%20} HTTP/1.1" 500 164 "-" "PostmanRuntime/7.26.10" 865

There are 2 problems. The first one is the error, the second is the unpredictable response time when using the WHERE filter. With a small data set there is no problem with either of them.