Cassandra integration creates too many sessions

See original GitHub issue

Describe the bug

Currently we are unable to enable the Cassandra integration for the OTEL agent due to the following error

{"thread":"ratpack-blocking-3-1","level":"WARN","loggerName":"com.datastax.oss.driver.internal.core.session.DefaultSession","message":"You have too many session instances: 143 active, expected less than 1 (see 'advanced.session-leak.threshold' in the configuration)","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","instant":{"epochSecond":1622035734,"nanoOfSecond":246582000},"contextMap":{"span_id":"f6a9043d82c4ca6a","trace_flags":"01","trace_id":"e9a3c7eb1ecf6c713cdc5cfe13ffc37f"},"threadId":400,"threadPriority":5}

This error will basically just keep being emitted(incrementing the # of sessions each time)

Setting OTEL_INSTRUMENTATION_CASSANDRA_ENABLED=false prevents the session issue, but may be causing a loss in context propegation

See this for more information on that propegation loss

Steps to reproduce

OTEL_INSTRUMENTATION_CASSANDRA_ENABLED=true

What did you expect to see? cassandra integration should emit spans correctly and not open so many sessions

What did you see instead?

{"thread":"ratpack-blocking-3-1","level":"WARN","loggerName":"com.datastax.oss.driver.internal.core.session.DefaultSession","message":"You have too many session instances: 143 active, expected less than 1 (see 'advanced.session-leak.threshold' in the configuration)","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","instant":{"epochSecond":1622035734,"nanoOfSecond":246582000},"contextMap":{"span_id":"f6a9043d82c4ca6a","trace_flags":"01","trace_id":"e9a3c7eb1ecf6c713cdc5cfe13ffc37f"},"threadId":400,"threadPriority":5}

What version are you using? Doesn’t matter which version of the agent, the issue is the same I have used the latest release(v1.2.0) as well as 1.1.0, 1.0.1 and 0.17.0

Environment Compiler: AdoptOpenJDK 11.0.6

Additional context

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:7 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
necrolyte2commented, Jun 4, 2021

I can try. Some proprietary code that might also be the cause so I’ll try to build a repo that enables cassandra without that code and see if it still happens.

0reactions
necrolyte2commented, Aug 9, 2021

I think I have gotten it down to the bare minimal and have it reproduced finally. Somehow the actual app we have starts up with the same error produced in the code below and then when a CQL session is initiated it generates all those errors like I posted originally.

The example app doesn’t even start up, but does generate the same error at startup time

REPRO: https://github.com/necrolyte2/ratpack-kotlin-otel/releases/tag/5.0.0

Read more comments on GitHub >

github_iconTop Results From Across the Web

How can I debug the error "too many continuous paging ...
ALLOW FILTERING: Invalid request, too many continuous paging sessions are already running: 60; at com.datastax.spark.connector.rdd.
Read more >
Creating a session takes too long when a Cassandra node is ...
Another possible workaround is to disable the session warmup [2] and create a similar process "manually" by sending a simple query with the...
Read more >
[JAVA-256] Too many files open - [Jira] - DataStax
I created 100 sessions (one session per keyspace); and they are ... Performance imapct; i ran into "too many files open" and the...
Read more >
datastax - Is creating a Cassandra session per client ok for low ...
It's better to share the session inside the one process, but if you have multiple separate processes, then it's ok to have separate...
Read more >
Manage Cassandra Connections - TechDocs
Manage Cassandra Connections. Last Updated July 2, 2020 ; maxConnectionsPerRemoteHost. The maximum number of connections per remote host. Default:.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found