Runtime context fails to detach token

See original GitHub issue

Describe your environment python 3.7 ujson sanic==20.9.1 opentelemetry-api==1.9.1 opentelemetry-sdk==1.9.1 opentelemetry-propagator-jaeger==1.9.1 opentelemetry-exporter-jaeger-thrift==1.9.1 opentelemetry-instrumentation==0.28b1 opentelemetry-exporter-otlp-proto-http==1.10.0

Steps to reproduce i have used the opentelemetry-instrumentation to create a middleware for sanic web framework, this middleware allows to trace a request, all seems good but sometimes the error Failed to detach context comes randomly for some request.

What is the expected behavior? instead of Failed to detach context error , original exception should be logged as exception so that the actual issue can be debugged with proper stacktrace etc. opentelemetry-python/opentelemetry-api/src/opentelemetry/context/init.py line here instead of

except Exception:  # pylint: disable=broad-except
        logger.error("Failed to detach context")

it should be

except Exception as e:  # pylint: disable=broad-except
        logger.error(e)

What is the actual behavior? getting Failed to detach context error message instead of original message.

Additional context

Issue Analytics

  • State:open
  • Created a year ago
  • Reactions:6
  • Comments:19 (9 by maintainers)

github_iconTop GitHub Comments

4reactions
kasiumcommented, Jun 1, 2022

I have the same request. Maybe the code could also be changed to

except Exception as e:  # pylint: disable=broad-except
        logger.exception("Failed to detach context")

Is a PR possible?

0reactions
morrissimocommented, Nov 29, 2022

server: FastAPI (uvicorn with uvloop) database: Postgres (peewee + psycopg2)

Chiming in with yet another observed instance of this behavior. We’re instrumenting a FastAPI app and first started seeing this Failed to detach context exception when we tweaked our database connection generator from using the start_as_current_span() decorator approach to instrumenting the method’s logic with a with block (+ events). FWIW, this exception being raised does NOT appear to prevent spans from being delivered to our OTEL endpoint 🤷

original method (does NOT cause the exception):

@tracer.start_as_current_span("get_db")
def get_db(db_state: Any = Depends(reset_db_state)) -> Generator[None, None, None]:
    with db.connection_context():
        yield

tweaked method (DOES cause the exception):

def get_db(db_state: Any = Depends(reset_db_state)) -> Generator[None, None, None]:
    with tracer.start_as_current_span("get_db") as span:
        span.add_event(name="entry", attributes=dict(db_is_closed=db.is_closed()))
        with db.connection_context():
            span.add_event(name="in context, pre-yield", attributes=dict(db_is_closed=db.is_closed()))
            yield
            span.add_event(name="in context, post-yield", attributes=dict(db_is_closed=db.is_closed()))
        span.add_event(name="exit", attributes=dict(db_is_closed=db.is_closed()))

The original motivation for the instrumentation change was when we noticed that we weren’t getting full spans using the decorator approach, and we wanted to add some events around the method’s yield to help us understand database connection timing.

Since trying out the tweaked method and observing this exception, we’ve tried a few different things:

  1. removing the span events: unsurprisingly, this made no difference - the exception was still raised
  2. using the decorator approach, and then grabbing the current span inside the method so the span events could be added: reverted to the original instrumentation behavior (the full span wasn’t captured, and no events were captured either), but the exception stopped being thrown as well
Read more comments on GitHub >

github_iconTop Results From Across the Web

Runtime context - AWS Cloud Development Kit (AWS CDK) v2
Context values that are set by the AWS CDK Toolkit (the cdk command) can be set automatically, from a file, or from the...
Read more >
Context token is in the future - SAP Community
If this query parameter is appended the runtime will check if this token is a valid timestamp and is not sent again in...
Read more >
Syntax Error in Angular App: Unexpected token - Stack Overflow
Yesterday I noticed that the main[hash].js was missing on the server and that Filezilla did not give me an error copying it.
Read more >
Hadoop Delegation Tokens Explained - Cloudera Blog
Remember in the 'Delegation Tokens at the Server-side' section, we explained the server has a background thread to remove expired tokens. So if ......
Read more >
Using Contexts - CircleCI
Restrict a context. CircleCI enables you to restrict secret environment variables at run time by adding security groups to contexts. Only organization ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found