[BUG] Cosmos SDK raises an exception when using the cosmos emulator

Describe the bug

The SDK throws an exception under the following conditions

• Using Cosmos Emulator (It works correctly in the real Cosmos DB)
• Data size is about 20 KB or more.
• Direct mode connection(not gateway mode)

There are no exceptions when you are in gateway mode or size is reduced. I ran similar code with the NET SDK and it was fine, so I don’t know if it’s a simulator problem or an SDK problem.

Exception or Stack Trace

Exception in thread "main" ServiceUnavailableException{userAgent=azsdk-java-cosmos/4.6.0 Windows10/10.0 JRE/11.0.8, error=null, resourceAddress='null', requestUri='null', statusCode=503, message=Service is currently unavailable, please retry after a while. If this problem persists please contact support.: Message: ChannelHandlerContext(RntbdRequestManager#0, [id: 0x1a9ab16c, L:0.0.0.0/0.0.0.0:6975]) closed exceptionally with 1 pending requests, {"userAgent":"azsdk-java-cosmos/4.6.0 Windows10/10.0 JRE/11.0.8","requestLatencyInMs":31062,"requestStartTimeUTC":"2020-10-13T01:00:58.187362100Z","requestEndTimeUTC":"2020-10-13T01:01:29.249632500Z","connectionMode":"DIRECT","responseStatisticsList":[{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"exception":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x35293730, L:0.0.0.0/0.0.0.0:6960]) closed exceptionally with 1 pending requests","transportRequestTimeline":[{"eventName":"created","durationInMicroSec":"60579","startTime":"2020-10-13T01:00:58.413362800Z"},{"eventName":"queued","durationInMicroSec":"121519","startTime":"2020-10-13T01:00:58.473942100Z"},{"eventName":"pipelined","durationInMicroSec":"29999","startTime":"2020-10-13T01:00:58.595461700Z"},{"eventName":"transitTime","durationInMicroSec":"10001","startTime":"2020-10-13T01:00:58.625460800Z"},{"eventName":"received","durationInMicroSec":"0","startTime":null},{"eventName":"completed","durationInMicroSec":"0","startTime":"2020-10-13T01:00:58.635462600Z"}],"rntbdRequestLengthInBytes":19438,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":19059,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":0,"pendingRequestsCount":0,"serviceEndpointStatistics":{"availableChannels":0,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2020-10-13T01:00:58.436Z","lastRequestTime":"2020-10-13T01:00:58.436Z","createdTime":"2020-10-13T01:00:58.420363800Z","isClosed":false}},"requestResponseTimeUTC":"2020-10-13T01:00:58.640460600Z","requestResourceType":"Document","requestOperationType":"Upsert"},{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"exception":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0xaf07c95b, L:0.0.0.0/0.0.0.0:6961]) closed exceptionally with 1 pending requests","transportRequestTimeline":[{"eventName":"created","durationInMicroSec":"0","startTime":"2020-10-13T01:00:58.653463800Z"},{"eventName":"queued","durationInMicroSec":"12000","startTime":"2020-10-13T01:00:58.653463800Z"},{"eventName":"pipelined","durationInMicroSec":"1997","startTime":"2020-10-13T01:00:58.665464500Z"},{"eventName":"transitTime","durationInMicroSec":"1012","startTime":"2020-10-13T01:00:58.667462100Z"},{"eventName":"received","durationInMicroSec":"0","startTime":null},{"eventName":"completed","durationInMicroSec":"0","startTime":"2020-10-13T01:00:58.668474200Z"}],"rntbdRequestLengthInBytes":19438,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":19059,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":1,"pendingRequestsCount":1,"serviceEndpointStatistics":{"availableChannels":0,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2020-10-13T01:00:58.436Z","lastRequestTime":"2020-10-13T01:00:58.414Z","createdTime":"2020-10-13T01:00:58.420363800Z","isClosed":false}},"requestResponseTimeUTC":"2020-10-13T01:00:58.668474200Z","requestResourceType":"Document","requestOperationType":"Upsert"},{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"exception":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0xe6e6c9b3, L:0.0.0.0/0.0.0.0:6962]) closed exceptionally with 1 pending requests","transportRequestTimeline":[{"eventName":"created","durationInMicroSec":"0","startTime":"2020-10-13T01:00:59.686463900Z"},{"eventName":"queued","durationInMicroSec":"14999","startTime":"2020-10-13T01:00:59.686463900Z"},{"eventName":"pipelined","durationInMicroSec":"1996","startTime":"2020-10-13T01:00:59.701463600Z"},{"eventName":"transitTime","durationInMicroSec":"1000","startTime":"2020-10-13T01:00:59.703460400Z"},{"eventName":"received","durationInMicroSec":"0","startTime":null},{"eventName":"completed","durationInMicroSec":"0","startTime":"2020-10-13T01:00:59.704461200Z"}],"rntbdRequestLengthInBytes":19438,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":19059,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":1,"pendingRequestsCount":1,"serviceEndpointStatistics":{"availableChannels":0,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2020-10-13T01:00:58.436Z","lastRequestTime":"2020-10-13T01:00:58.653Z","createdTime":"2020-10-13T01:00:58.420363800Z","isClosed":false}},"requestResponseTimeUTC":"2020-10-13T01:00:59.704461200Z","requestResourceType":"Document","requestOperationType":"Upsert"},{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"exception":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x41dd827c, L:0.0.0.0/0.0.0.0:6963]) closed exceptionally with 1 pending requests","transportRequestTimeline":[{"eventName":"created","durationInMicroSec":"0","startTime":"2020-10-13T01:01:01.722464200Z"},{"eventName":"queued","durationInMicroSec":"13998","startTime":"2020-10-13T01:01:01.722464200Z"},{"eventName":"pipelined","durationInMicroSec":"1000","startTime":"2020-10-13T01:01:01.736462700Z"},{"eventName":"transitTime","durationInMicroSec":"2080","startTime":"2020-10-13T01:01:01.737462800Z"},{"eventName":"received","durationInMicroSec":"0","startTime":null},{"eventName":"completed","durationInMicroSec":"0","startTime":"2020-10-13T01:01:01.739543300Z"}],"rntbdRequestLengthInBytes":19438,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":19059,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":1,"pendingRequestsCount":1,"serviceEndpointStatistics":{"availableChannels":0,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2020-10-13T01:00:58.436Z","lastRequestTime":"2020-10-13T01:00:59.686Z","createdTime":"2020-10-13T01:00:58.420363800Z","isClosed":false}},"requestResponseTimeUTC":"2020-10-13T01:01:01.739543300Z","requestResourceType":"Document","requestOperationType":"Upsert"},{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"exception":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0xe8ee701f, L:0.0.0.0/0.0.0.0:6966]) closed exceptionally with 1 pending requests","transportRequestTimeline":[{"eventName":"created","durationInMicroSec":"0","startTime":"2020-10-13T01:01:05.750169500Z"},{"eventName":"queued","durationInMicroSec":"13996","startTime":"2020-10-13T01:01:05.750169500Z"},{"eventName":"pipelined","durationInMicroSec":"2003","startTime":"2020-10-13T01:01:05.764165600Z"},{"eventName":"transitTime","durationInMicroSec":"1011","startTime":"2020-10-13T01:01:05.766169300Z"},{"eventName":"received","durationInMicroSec":"0","startTime":null},{"eventName":"completed","durationInMicroSec":"0","startTime":"2020-10-13T01:01:05.767181200Z"}],"rntbdRequestLengthInBytes":19438,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":19059,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":1,"pendingRequestsCount":1,"serviceEndpointStatistics":{"availableChannels":0,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2020-10-13T01:00:58.436Z","lastRequestTime":"2020-10-13T01:01:01.722Z","createdTime":"2020-10-13T01:00:58.420363800Z","isClosed":false}},"requestResponseTimeUTC":"2020-10-13T01:01:05.767181200Z","requestResourceType":"Document","requestOperationType":"Upsert"},{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"exception":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0xc88a5235, L:0.0.0.0/0.0.0.0:6973]) closed exceptionally with 1 pending requests","transportRequestTimeline":[{"eventName":"created","durationInMicroSec":"0","startTime":"2020-10-13T01:01:13.776169300Z"},{"eventName":"queued","durationInMicroSec":"14998","startTime":"2020-10-13T01:01:13.776169300Z"},{"eventName":"pipelined","durationInMicroSec":"1999","startTime":"2020-10-13T01:01:13.791168Z"},{"eventName":"transitTime","durationInMicroSec":"997","startTime":"2020-10-13T01:01:13.793167600Z"},{"eventName":"received","durationInMicroSec":"0","startTime":null},{"eventName":"completed","durationInMicroSec":"0","startTime":"2020-10-13T01:01:13.794165400Z"}],"rntbdRequestLengthInBytes":19438,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":19059,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":1,"pendingRequestsCount":1,"serviceEndpointStatistics":{"availableChannels":0,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2020-10-13T01:00:58.436Z","lastRequestTime":"2020-10-13T01:01:05.750Z","createdTime":"2020-10-13T01:00:58.420363800Z","isClosed":false}},"requestResponseTimeUTC":"2020-10-13T01:01:13.794165400Z","requestResourceType":"Document","requestOperationType":"Upsert"},{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"exception":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x1a9ab16c, L:0.0.0.0/0.0.0.0:6975]) closed exceptionally with 1 pending requests","transportRequestTimeline":[{"eventName":"created","durationInMicroSec":"0","startTime":"2020-10-13T01:01:29.224676800Z"},{"eventName":"queued","durationInMicroSec":"19765","startTime":"2020-10-13T01:01:29.224676800Z"},{"eventName":"pipelined","durationInMicroSec":"3092","startTime":"2020-10-13T01:01:29.244442600Z"},{"eventName":"transitTime","durationInMicroSec":"986","startTime":"2020-10-13T01:01:29.247535Z"},{"eventName":"received","durationInMicroSec":"0","startTime":null},{"eventName":"completed","durationInMicroSec":"1110","startTime":"2020-10-13T01:01:29.248521800Z"}],"rntbdRequestLengthInBytes":19438,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":19059,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":1,"pendingRequestsCount":1,"serviceEndpointStatistics":{"availableChannels":0,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2020-10-13T01:00:58.436Z","lastRequestTime":"2020-10-13T01:01:13.776Z","createdTime":"2020-10-13T01:00:58.420363800Z","isClosed":false}},"requestResponseTimeUTC":"2020-10-13T01:01:29.249632500Z","requestResourceType":"Document","requestOperationType":"Upsert"}],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{"8e25540e-0cef-11eb-88e4-776c31486ee2":{"startTimeUTC":"2020-10-13T01:00:58.646461900Z","endTimeUTC":"2020-10-13T01:00:58.651463700Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2FG4RhAA%3D%3D%2Fcolls%2FG4RhAKbZZTs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","errorMessage":null,"inflightRequest":false},"8dfd0b8c-0cef-11eb-88e4-776c31486ee2":{"startTimeUTC":"2020-10-13T01:00:58.382365200Z","endTimeUTC":"2020-10-13T01:00:58.402363Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2FG4RhAA%3D%3D%2Fcolls%2FG4RhAKbZZTs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","errorMessage":null,"inflightRequest":false},"97295f76-0cef-11eb-88e4-776c31486ee2":{"startTimeUTC":"2020-10-13T01:01:13.772228500Z","endTimeUTC":"2020-10-13T01:01:13.775168800Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2FG4RhAA%3D%3D%2Fcolls%2FG4RhAKbZZTs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","errorMessage":null,"inflightRequest":false},"a01faf38-0cef-11eb-88e4-776c31486ee2":{"startTimeUTC":"2020-10-13T01:01:28.808897500Z","endTimeUTC":"2020-10-13T01:01:29.223637500Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2FG4RhAA%3D%3D%2Fcolls%2FG4RhAKbZZTs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","errorMessage":null,"inflightRequest":false},"8ff9ed02-0cef-11eb-88e4-776c31486ee2":{"startTimeUTC":"2020-10-13T01:01:01.717498700Z","endTimeUTC":"2020-10-13T01:01:01.721463700Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2FG4RhAA%3D%3D%2Fcolls%2FG4RhAKbZZTs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","errorMessage":null,"inflightRequest":false},"92608cc4-0cef-11eb-88e4-776c31486ee2":{"startTimeUTC":"2020-10-13T01:01:05.745194800Z","endTimeUTC":"2020-10-13T01:01:05.749191500Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2FG4RhAA%3D%3D%2Fcolls%2FG4RhAKbZZTs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","errorMessage":null,"inflightRequest":false},"8ec341c0-0cef-11eb-88e4-776c31486ee2":{"startTimeUTC":"2020-10-13T01:00:59.681496200Z","endTimeUTC":"2020-10-13T01:00:59.685464700Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2FG4RhAA%3D%3D%2Fcolls%2FG4RhAKbZZTs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","errorMessage":null,"inflightRequest":false}},"regionsContacted":["https://127.0.0.1:8081/"],"retryContext":{"retryCount":6,"statusAndSubStatusCodes":[[410,0],[410,0],[410,0],[410,0],[410,0],[410,0],[410,0]],"retryLatency":30609},"metadataDiagnosticsContext":{"metadataDiagnosticList":[{"metaDataName":"CONTAINER_LOOK_UP","startTimeUTC":"2020-10-13T01:00:58.238363400Z","endTimeUTC":"2020-10-13T01:00:58.260362600Z","durationinMS":21},{"metaDataName":"PARTITION_KEY_RANGE_LOOK_UP","startTimeUTC":"2020-10-13T01:00:58.309365500Z","endTimeUTC":"2020-10-13T01:00:58.344367900Z","durationinMS":35},{"metaDataName":"SERVER_ADDRESS_LOOKUP","startTimeUTC":"2020-10-13T01:00:58.382365200Z","endTimeUTC":"2020-10-13T01:00:58.402363Z","durationinMS":19},{"metaDataName":"SERVER_ADDRESS_LOOKUP","startTimeUTC":"2020-10-13T01:00:58.646461900Z","endTimeUTC":"2020-10-13T01:00:58.651463700Z","durationinMS":5},{"metaDataName":"SERVER_ADDRESS_LOOKUP","startTimeUTC":"2020-10-13T01:00:59.681496200Z","endTimeUTC":"2020-10-13T01:00:59.685464700Z","durationinMS":3},{"metaDataName":"SERVER_ADDRESS_LOOKUP","startTimeUTC":"2020-10-13T01:01:01.717498700Z","endTimeUTC":"2020-10-13T01:01:01.721463700Z","durationinMS":3},{"metaDataName":"SERVER_ADDRESS_LOOKUP","startTimeUTC":"2020-10-13T01:01:05.745194800Z","endTimeUTC":"2020-10-13T01:01:05.749191500Z","durationinMS":3},{"metaDataName":"SERVER_ADDRESS_LOOKUP","startTimeUTC":"2020-10-13T01:01:13.772228500Z","endTimeUTC":"2020-10-13T01:01:13.775168800Z","durationinMS":2},{"metaDataName":"SERVER_ADDRESS_LOOKUP","startTimeUTC":"2020-10-13T01:01:28.808897500Z","endTimeUTC":"2020-10-13T01:01:29.223637500Z","durationinMS":414}]},"serializationDiagnosticsContext":{"serializationDiagnosticsList":[{"serializationType":"ITEM_SERIALIZATION","startTimeUTC":"2020-10-13T01:00:58.198366600Z","endTimeUTC":"2020-10-13T01:00:58.228361700Z","durationInMicroSec":29995},{"serializationType":"PARTITION_KEY_FETCH_SERIALIZATION","startTimeUTC":"2020-10-13T01:00:58.269363700Z","endTimeUTC":"2020-10-13T01:00:58.275361500Z","durationInMicroSec":5997}]},"gatewayStatistics":null,"systemInformation":{"usedMemory":"34379 KB","availableMemory":"8292789 KB","systemCpuLoad":"(2020-10-13T01:01:02.144118800Z 13.1%), (2020-10-13T01:01:07.139199900Z 10.2%), (2020-10-13T01:01:12.135200900Z 10.7%), (2020-10-13T01:01:17.146212600Z 10.7%), 
(2020-10-13T01:01:22.142164300Z 12.1%), (2020-10-13T01:01:27.140573600Z 12.0%)"},"clientCfgs":{"id":0,"numberOfClients":1,"connCfg":{"rntbd":"(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30)","gw":"(cps:1000, rto:PT5S, icto:null, p:false)","other":"(ed: true, cs: false)"},"consistencyCfg":"(consistency: null, 
mm: true, prgns: [])"}}, causeInfo=[class: class com.azure.cosmos.implementation.GoneException, message: ChannelHandlerContext(RntbdRequestManager#0, [id: 0x1a9ab16c, L:0.0.0.0/0.0.0.0:6975]) closed exceptionally with 1 pending requests], responseHeaders={}, requestHeaders=null}
        at com.azure.cosmos.BridgeInternal.createServiceUnavailableException(BridgeInternal.java:588)
        at com.azure.cosmos.implementation.directconnectivity.GoneAndRetryWithRetryPolicy.shouldRetry(GoneAndRetryWithRetryPolicy.java:87)
        at com.azure.cosmos.implementation.RetryUtils.lambda$toRetryWithAlternateFunc$4(RetryUtils.java:76)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88)
        at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:181)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185)
        at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251)
        at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:251)
        at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onError(FluxContextStart.java:110)
        at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onError(FluxDoFinally.java:129)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:100)
        at reactor.core.publisher.Operators.error(Operators.java:196)
        at reactor.core.publisher.MonoError.subscribe(MonoError.java:52)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:97)
        at reactor.core.publisher.MonoCompletionStage.lambda$subscribe$0(MonoCompletionStage.java:80)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
        at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.completeAllPendingRequestsExceptionally(RntbdRequestManager.java:677)     
        at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.channelUnregistered(RntbdRequestManager.java:255)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:198)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:184)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:177)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelUnregistered(CombinedChannelDuplexHandler.java:406)
        at io.netty.channel.CombinedChannelDuplexHandler.channelUnregistered(CombinedChannelDuplexHandler.java:203)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:198)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:184)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:177)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1388)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:198)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:184)
        at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:821)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:826)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
                at reactor.core.publisher.Mono.block(Mono.java:1680)
                at com.azure.cosmos.CosmosContainer.blockItemResponse(CosmosContainer.java:230)
                at com.azure.cosmos.CosmosContainer.upsertItem(CosmosContainer.java:194)
                at com.example.App.main(App.java:45)
Caused by: GoneException{userAgent=azsdk-java-cosmos/4.6.0 Windows10/10.0 JRE/11.0.8, error=null, resourceAddress='rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer11/partitions/a4cb4957-38c8-11e6-8106-8cdcd42c33be/replicas/1p/', requestUri='null', statusCode=410, message=ChannelHandlerContext(RntbdRequestManager#0, [id: 0x1a9ab16c, L:0.0.0.0/0.0.0.0:6975]) closed exceptionally with 1 pending requests, causeInfo=[class: class java.nio.channels.ClosedChannelException, message: null], responseHeaders={}, requestHeaders=[Accept=application/json, x-ms-date=Tue, 13 Oct 2020 01:00:58 GMT, x-ms-documentdb-collection-rid=G4RhAKbZZTs=, x-ms-client-retry-attempt-count=6, Prefer=return=minimal, x-ms-documentdb-partitionkey=["e0a3bb55-94d6-411a-93d0-a595e2c36365"], x-ms-documentdb-is-upsert=true, x-ms-remaining-time-in-ms-on-client=15000, Content-Type=application/json]}
        at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.completeAllPendingRequestsExceptionally(RntbdRequestManager.java:674)     
        ... 21 more
Caused by: java.nio.channels.ClosedChannelException
        at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.channelUnregistered(Unknown Source)

To Reproduce

• Setup Azure Cosmos DB Emulator
• Add certificate to Java keystore
• Execute following code.

Code Snippet

It runs normally when gateway mode is enabled or the size is reduced.

package com.example;

import java.io.IOException;
import java.util.UUID;

import com.azure.cosmos.CosmosClient;
import com.azure.cosmos.CosmosClientBuilder;
import com.azure.cosmos.CosmosContainer;
import com.azure.cosmos.CosmosDatabase;
import com.fasterxml.jackson.databind.ObjectMapper;

public class App {

    // Emulator
    private static String HOST = "https://localhost:8081/";
    private static String KEY = "your emulators key"; 
    private static String DATABASE = "testdb";
    private static String CONTAINER = "container";

    public static void main(String[] args) throws IOException {

        int length = 20000;

        CosmosClient client = new CosmosClientBuilder()
                .endpoint(HOST)
                .key(KEY)
                //.gatewayMode()
                .buildClient();

        CosmosDatabase db = client.getDatabase(DATABASE);
        CosmosContainer container = db.getContainer(CONTAINER);

        // init data
        SampleData data = new SampleData();
        data.setId(UUID.randomUUID().toString());
        data.setMessages("a".repeat(length));

        // check data size
        ObjectMapper mapper = new ObjectMapper();
        byte[] bytes = mapper.writeValueAsBytes(data);
        System.out.format("Data size = %d\n", bytes.length);

        // upsert item
        var res = container.upsertItem(data);
        System.out.format("Status code = %d\n", res.getStatusCode());

        client.close();
    }

    public static class SampleData {

        private String id;
        private String message;

        public String getId() {
            return id;
        }

        public void setId(String id) {
            this.id = id;
        }

        public String getMessages() {
            return message;
        }

        public void setMessages(String message) {
            this.message = message;
        }
    }

}

Expected behavior It works normally without any exceptions.

Setup (please complete the following information):

• OS: Windows
• IDE : VS Code
• Java : java version “11.0.8” 2020-07-14 LTS
• Cosmos SDK : 4.6.0
• Cosmos Emulator : 2.11.6 (6 October 2020)

Additional context Add any other context about the problem here.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added