Description

Intermittently seeing “Confluent.Kafka.KafkaException Local: Queue Full”

How to reproduce

We are unable to reproduce locally. We only see this in long running processes (IIS hosted Web Apps) Restarting the application pool fixes the problem. We see this problem around every 3 days but there is no clear pattern and it is not happening after App Pool restarts. During these 3(ish) days, we would have sent roughly 210K messages. We have the queue.buffering.max.messages and queue.buffering.max.kbytes set at their defaults of 100K and 400K respectively.

We don’t loose any messages until the Queue Full exceptions start. I would guess the local queue is filling with delivery reports?

Is it possible that the long running task in the Producer that calls rd_kafka_poll is dying somehow? We set manualPoll = false on construction of the Producer.

We do not see any exceptions pertaining to the kafka producer at all ever except Local: Queue Full. We capture all Unobserved exceptions in a global handler for logging.

We are not producing a high volume of msgs/s. Only around 13/s

Checklist

Please provide the following information:

• Confluent.Kafka nuget version: 0.9.4

• Apache Kafka version: 0.10.1.1-cp1

• Client configuration: Global: api.version.request: true Topic: request.required.acks: 0 request.timeout.ms: 5000 message.timeout.ms: 2000

• Operating system: Windows 2012

• Provide logs (with “debug” : “…” as necessary in configuration)

Confluent.Kafka.KafkaException: Local: Queue full
   at Confluent.Kafka.Producer.Produce(String topic, Byte[] val, Int32 valOffset, Int32 valLength, Byte[] key, Int32 keyOffset, Int32 keyLength, Nullable`1 timestamp, Int32 partition, Boolean blockIfQueueFull, IDeliveryHandler deliveryHandler)
   at Confluent.Kafka.Producer.Produce(String topic, Byte[] val, Int32 valOffset, Int32 valLength, Byte[] key, Int32 keyOffset, Int32 keyLength, Nullable`1 timestamp, Nullable`1 partition, Boolean blockIfQueueFull)
   at Confluent.Kafka.Producer.ProduceAsync(String topic, Byte[] key, Byte[] val)

• Provide broker log excerpts
• Critical issue