Trace

[!TIP] This document is machine-translated by Google. If you find grammatical and semantic errors, and the document description is not clear, please PR

Foreword

In the microservice architecture, the call chain may be very long, from http to rpc, and from rpc to http. Developers want to know the call status and performance of each link, the best solution is full link tracking.

The tracking method is to generate its own spanID at the beginning of a request, and pass it down along the entire request link. We use this spanID to view the status of the entire link and performance issues.

Let’s take a look at the link implementation of go-zero.

Code structure

  • spancontext ：保存链路的上下文信息「traceid，spanid，或者是其他想要传递的内容」
  • span ：链路中的一个操作，存储时间和某些信息
  • propagator ： trace 传播下游的操作「抽取，注入」
  • noop ：实现了空的 tracer 实现

Trace - 图1

Concept

SpanContext

Before introducing span, first introduce context. SpanContext saves the context information of distributed tracing, including Trace id, Span id and other content that needs to be passed downstream. The implementation of OpenTracing needs to pass the SpanContext through a certain protocol to associate the Span in different processes to the same Trace. For HTTP requests, SpanContext is generally passed using HTTP headers.

Below is the spanContext implemented by go-zero by default

  1. type spanContext struct {
  2.     traceId string      // TraceID represents the globally unique ID of tracer
  3.     spanId  string      // SpanId indicates the unique ID of a span in a single trace, which is unique in the trace
  4. }

At the same time, developers can also implement the interface methods provided by SpanContext to realize their own contextual information transfer:

  1. type SpanContext interface {
  2.     TraceId() string                        // get TraceId
  3.     SpanId() string                         // get SpanId
  4.     Visit(fn func(key, val string) bool)    // Custom operation TraceId, SpanId
  5. }

Span

A REST call or database operation, etc., can be used as a span. span is the smallest tracking unit of distributed tracing. A trace is composed of multiple spans. The tracking information includes the following information:

  1. type Span struct {
  2.     ctx           spanContext       
  3.     serviceName   string           
  4.     operationName string           
  5.     startTime     time.Time         
  6.     flag          string           
  7.     children      int              
  8. }

Judging from the definition structure of span: In microservices, this is a complete sub-calling process, with the start of the call startTime, the context structure spanContext that marks its own unique attribute, and the number of child nodes of fork.

Example application

In go-zero, http and rpc have been integrated as built-in middleware. We use http, rpc, take a look at how tracing is used:

HTTP

  1. func TracingHandler(next http.Handler) http.Handler {
  2.     return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  3.         // **1**
  4.         carrier, err := trace.Extract(trace.HttpFormat, r.Header)
  5.         // ErrInvalidCarrier means no trace id was set in http header
  6.         if err != nil && err != trace.ErrInvalidCarrier {
  7.             logx.Error(err)
  8.         }
  10.         // **2**
  11.         ctx, span := trace.StartServerSpan(r.Context(), carrier, sysx.Hostname(), r.RequestURI)
  12.         defer span.Finish()
  13.         // **5**
  14.         r = r.WithContext(ctx)
  16.         next.ServeHTTP(w, r)
  17.     })
  18. }
  20. func StartServerSpan(ctx context.Context, carrier Carrier, serviceName, operationName string) (
  21.     context.Context, tracespec.Trace) {
  22.     span := newServerSpan(carrier, serviceName, operationName)
  23.     // **4**
  24.     return context.WithValue(ctx, tracespec.TracingKey, span), span
  25. }
  27. func newServerSpan(carrier Carrier, serviceName, operationName string) tracespec.Trace {
  28.     // **3**
  29.     traceId := stringx.TakeWithPriority(func() string {
  30.         if carrier != nil {
  31.             return carrier.Get(traceIdKey)
  32.         }
  33.         return ""
  34.     }, func() string {
  35.         return stringx.RandId()
  36.     })
  37.     spanId := stringx.TakeWithPriority(func() string {
  38.         if carrier != nil {
  39.             return carrier.Get(spanIdKey)
  40.         }
  41.         return ""
  42.     }, func() string {
  43.         return initSpanId
  44.     })
  46.     return &Span{
  47.         ctx: spanContext{
  48.             traceId: traceId,
  49.             spanId:  spanId,
  50.         },
  51.         serviceName:   serviceName,
  52.         operationName: operationName,
  53.         startTime:     timex.Time(),
  54.         // 标记为server
  55.         flag:          serverFlag,
  56.     }
  57. }
  1. Set header -> carrier to get the traceId and other information in the header
  2. Open a new span and encapsulate “traceId, spanId” in the context
  3. Obtain traceId and spanId from the aforementioned carrier “that is, header” -See if it is set in the header -If it is not set, it will be randomly generated and returned
  4. Generate a new ctx from request, encapsulate the corresponding information in ctx, and return
  5. From the above context, copy a copy to the current request

Trace - 图2

In this way, the information of the span is passed to the downstream service along with the request.

RPC

There are client, server in rpc, so from tracing there are also clientTracing, serverTracing. The logic of serveTracing is basically the same as that of http. Let’s take a look at how clientTracing is used?

  1. func TracingInterceptor(ctx context.Context, method string, req, reply interface{},
  2.     cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
  3.     // open clientSpan
  4.     ctx, span := trace.StartClientSpan(ctx, cc.Target(), method)
  5.     defer span.Finish()
  7.     var pairs []string
  8.     span.Visit(func(key, val string) bool {
  9.         pairs = append(pairs, key, val)
  10.         return true
  11.     })
  12.     // **3** Add the data in the pair to ctx in the form of a map
  13.     ctx = metadata.AppendToOutgoingContext(ctx, pairs...)
  15.     return invoker(ctx, method, req, reply, cc, opts...)
  16. }
  18. func StartClientSpan(ctx context.Context, serviceName, operationName string) (context.Context, tracespec.Trace) {
  19.     // **1**
  20.     if span, ok := ctx.Value(tracespec.TracingKey).(*Span); ok {
  21.         // **2**
  22.         return span.Fork(ctx, serviceName, operationName)
  23.     }
  25.     return ctx, emptyNoopSpan
  26. }
  1. Get the span context information brought down by the upstream
  2. Create a new ctx from the acquired span, span “inherit the traceId of the parent span”
  3. Add the span generated data to ctx, pass it to the next middleware, and flow downstream

Summary

go-zero obtains the link traceID by intercepting the request, and then assigns a root Span at the entry of the middleware function, and then splits the child Spans in subsequent operations. Each span has its own specific identification. After Finsh Will be collected in the link tracking system. Developers can trace the traceID through the ELK tool to see the entire call chain.

At the same time, go-zero does not provide a complete set of trace link solutions. Developers can encapsulate the existing span structure of go-zero, build their own reporting system, and access links such as jaeger, zipkin, etc. Tracking tool.

Reference