[BUG] Service Bus Receiver stops receiving messages from queues after Server Errors

See original GitHub issue

Describe the bug When there is Server Errors in Service Bus, which happen once or twice per week. The Service Bus queue receiver stops receiving the messages, while the queue sender can keep sending them.

The only way we can fix this is to restart the app.

Exception or Stack Trace This is one of the errors after the Server Errors:

{"az.sdk.message":"Non-retryable error occurred in AMQP receive link.","exception":"The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:ebae50e8a3044adf88825a821bf53108_G28, SystemTracker:gateway7, Timestamp:2022-04-26T02:03:16, errorContext[NAMESPACE: asavie-platform-servicebus-stg-eu-1.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: my-queue-here, REFERENCE_ID: my-queue-here_7a1b06_1649943062396, LINK_CREDIT: 2]","linkName":"n/a","entityPath":"n/a"}

To Reproduce Create a receiver no matter if using the synchronous or the processor model and leave it running for a week receiving messages. Wait until there is Server Errors, shortly after that the Receiver stops receiving messages while the Sender can keep sending them.

The problem here is that not all the Server Errors cause this problem, but since they all arrive in bulk, one of them trigger the problem, here is a screenshot of the Server error that caused the problem:

On the top is this log occurrence, in the middle is the number of server errors and at the bottom is the messages in and out. You can see that is during the 4th server error that the app logs the error and stops receiving the messages. The yellow line is the received messages and the green line is the sent messages.

Code Snippet

this.serviceBusReceiver = new ServiceBusClientBuilder()
                .connectionString(applicationProperties.getServiceBusQueueConnectionString())
                .processor()
                .queueName(applicationProperties.getServiceBusQueueName())
                .processMessage(this::onMessage)
                .processError(ServiceBusQueueReceiver::handleError)
                .buildProcessorClient();

Expected behavior The Receiver should reconnect properly and keep receiving the messages.

Setup (please complete the following information):

OS: Linux
Library/Libraries: com.azure:azure-messaging-servicebus:7.7.0
Java version: 11
App Server/Environment: Tomcat
Frameworks: Spring Boot

Additional context We have multiple clusters with the same setup pointing to multiple Service Bus namespaces configured exactly the same way. All of them have the same issue, as soon as there are server errors in service bus, the receiver stops receiving messages.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

Bug Description Added
Repro Steps Added
Setup information Added

Issue Analytics

State:
Created a year ago
Reactions:1
Comments:18 (6 by maintainers)

Top GitHub Comments

3reactions

anuchandycommented, May 25, 2022

Hi Quentin, you’re right that when using “ServiceBusAsyncReceiver” you’ll have to handle the case where the receiver instance bubble up a non-retriable error or exhaust retry. Like I mentioned, “error is a terminal event by reactive contract”.

Depending on how much your application logic is integrated with the Reactor framework, the code pattern for recovery you want to use may differ; if we start with a very simplified apporach, it will be -

String connectionString = System.getenv(SERVICEBUS_CONNECTION_STRING);
String queueName = System.getenv(SERVICEBUS_QUEUE_NAME);

while (true) {
    // Using the "current thread" to create the client.
    try (ServiceBusReceiverAsyncClient client = new ServiceBusClientBuilder()
            .connectionString(connectionString)
            .receiver()
            .queueName(queueName)
            .receiveMode(ServiceBusReceiveMode.PEEK_LOCK)
            .buildAsyncClient()) {

        final CountDownLatch errorLatch = new CountDownLatch(1);

        client.receiveMessages()
                // messages and error are received on "boundedElastic thread".
                .flatMap(message -> {
                    return handleReceivedMessage(message);
                })
                .subscribe(
                    __ -> {},
                    error -> {
                        LOGGER.info("Retry Exhausted or Non-Retriable error happened: {}", error);
                        errorLatch.countDown();
                    });

        // block the "current thread" until there is an error.
        errorLatch.await();
        // latch unblocked, let's auto-close the current receiver, backoff and repeat.
    }

    // backoff
    TimeUnit.SECONDS.sleep(1);
}

Let me know how it goes.

If we want to go a more reactive way of recovery, then there is retryWhen operator

1reaction

quentinpoppcommented, May 10, 2022

We got the same problem. Any news?

Top Results From Across the Web

Troubleshooting guide for Azure Service Bus - Microsoft Learn

This article provides troubleshooting tips and recommendations for a few issues that you may see when using Azure Service Bus.

Handling Azure Service Bus errors with .NET - Code4IT

Senders and Receivers handle errors on Azure Service Bus differently. ... To receive messages from a Topic, we need the following steps:.

azure-servicebus - PyPI

The following sections provide several code snippets covering some of the most common Service Bus tasks, including: Send messages to a queue; Receive...

Azure ServiceBus errors in production log - Google Groups

We are running MassTransit in production with Azure ServiceBus (ASB) and we frequently get error logs related to the use of the Azure...

Azure Service Bus and its Complete Overview | Serverless360

That is, receivers typically receive and process messages in the order in which they were added to the queue, and only one message...