Python

The work of instrumentation libraries generally consists of three steps:

  1. When a service receives a new request (over HTTP or some other protocol), it uses OpenTracing’s inject/extract API to continue an active trace, creating a Span object in the process. If the request does not contain an active trace, the service starts a new trace and a new root Span.
  2. The service needs to store the current Span in some request-local storage, (called Span activation) where it can be retrieved from when a child Span must be created, e.g. in case of the service making an RPC to another service.
  3. When making outbound calls to another service, the current Span must be retrieved from request-local storage, a child span must be created (e.g., by using the start_child_span() helper), and that child span must be embedded into the outbound request (e.g., using HTTP headers) via OpenTracing’s inject/extract API.

Below are the code examples for the previously mentioned steps. Implementation of request-local storage needed for step 2 is specific to the service and/or frameworks / instrumentation libraries it is using, exposed as a ScopeManager child contained as Tracer.scope_manager. See details below.

Inbound request

Somewhere in your server’s request handler code:

  1. def handle_request(request):
  2. span = before_request(request, opentracing.global_tracer())
  3. # store span in some request-local storage using Tracer.scope_manager,
  4. # using the returned `Scope` as Context Manager to ensure
  5. # `Span` will be cleared and (in this case) `Span.finish()` be called.
  6. with tracer.scope_manager.activate(span, True) as scope:
  7. # actual business logic
  8. handle_request_for_real(request)
  9. def before_request(request, tracer):
  10. span_context = tracer.extract(
  11. format=Format.HTTP_HEADERS,
  12. carrier=request.headers,
  13. )
  14. span = tracer.start_span(
  15. operation_name=request.operation,
  16. child_of(span_context))
  17. span.set_tag('http.url', request.full_url)
  18. remote_ip = request.remote_ip
  19. if remote_ip:
  20. span.set_tag(tags.PEER_HOST_IPV4, remote_ip)
  21. caller_name = request.caller_name
  22. if caller_name:
  23. span.set_tag(tags.PEER_SERVICE, caller_name)
  24. remote_port = request.remote_port
  25. if remote_port:
  26. span.set_tag(tags.PEER_PORT, remote_port)
  27. return span

Outbound request

Somewhere in your service that’s about to make an outgoing call:

  1. from opentracing import tags
  2. from opentracing.propagation import Format
  3. from opentracing_instrumentation import request_context
  4. # create and serialize a child span and use it as context manager
  5. with before_http_request(
  6. request=out_request,
  7. current_span_extractor=request_context.get_current_span):
  8. # actual call
  9. return urllib2.urlopen(request)
  10. def before_http_request(request, current_span_extractor):
  11. op = request.operation
  12. parent_span = current_span_extractor()
  13. outbound_span = opentracing.global_tracer().start_span(
  14. operation_name=op,
  15. child_of=parent_span
  16. )
  17. outbound_span.set_tag('http.url', request.full_url)
  18. service_name = request.service_name
  19. host, port = request.host_port
  20. if service_name:
  21. outbound_span.set_tag(tags.PEER_SERVICE, service_name)
  22. if host:
  23. outbound_span.set_tag(tags.PEER_HOST_IPV4, host)
  24. if port:
  25. outbound_span.set_tag(tags.PEER_PORT, port)
  26. http_header_carrier = {}
  27. opentracing.global_tracer().inject(
  28. span_context=outbound_span,
  29. format=Format.HTTP_HEADERS,
  30. carrier=http_header_carrier)
  31. for key, value in http_header_carrier.iteritems():
  32. request.add_header(key, value)
  33. return outbound_span

Scope and within-process propagation

For getting/setting the current active Span in the used request-local storage, OpenTracing requires that every Tracer contains a ScopeManager that grants access to the active Span through a Scope. Any Span may be transferred to another task or thread, but not Scope.

  1. # Access to the active span is straightforward.
  2. scope = tracer.scope_manager.active()
  3. if scope is not None:
  4. scope.span.set_tag('...', '...')

The common case starts a Scope that’s automatically registered for intra-process propagation via ScopeManager.

Note that start_active_span('...') automatically finishes the span on Scope.close() (start_active_span('...', finish_on_close=False) does not finish it, in contrast).

  1. # Manual activation of the Span.
  2. span = tracer.start_span(operation_name='someWork')
  3. with tracer.scope_manager.activate(span, True) as scope:
  4. # Do things.
  5. # Automatic activation of the Span.
  6. # finish_on_close is a required parameter.
  7. with tracer.start_active_span('someWork', finish_on_close=True) as scope:
  8. # Do things.
  9. # Handling done through a try construct:
  10. span = tracer.start_span(operation_name='someWork')
  11. scope = tracer.scope_manager.activate(span, True)
  12. try:
  13. # Do things.
  14. except Exception as e:
  15. scope.set_tag('error', '...')
  16. finally:
  17. scope.finish()

If there is a Scope, it will act as the parent to any newly started Span unless the programmer passes ignore_active_span=True at start_span()/start_active_span() time or specified parent context explicitly:

  1. scope = tracer.start_active_span('someWork', ignore_active_span=True)

Each service/framework ought to provide a specific ScopeManager implementation that relies on their own request-local storage (thread-local storage, or coroutine-based storage for asynchronous frameworks, for example).

Scope managers

This project includes a set of ScopeManager implementations under the opentracing.scope_managers submodule, which can be imported on demand:

  1. from opentracing.scope_managers import ThreadLocalScopeManager

There exist implementations for thread-local (the default), gevent, Tornado and asyncio:

  1. from opentracing.scope_managers.gevent import GeventScopeManager # requires gevent
  2. from opentracing.scope_managers.tornado import TornadoScopeManager # requires Tornado
  3. from opentracing.scope_managers.asyncio import AsyncioScopeManager # requires Python 3.4 or newer.