Opentracing + Uber Jaeger full link gray call chain, Nepxion Discovery

Keywords: Programming github Spring Mobile Java

When the gateway and service are implementing the whole link distributed gray-scale publishing and routing, we need a tracking system to monitor which gray-scale group, which gray-scale version, which gray-scale area the gateway and service are going through, and even monitor the gray-scale rules and routing strategies that are passed from the head of Http Header. This function means:

It can not only monitor the basic call information in the whole link, but also monitor the additional gray information, which helps us to judge whether the gray publishing and routing are accurate. Once there is a problem, it can also locate quickly.
You can monitor when traffic is switched to a new version, or to a new area, or to a new machine
It can monitor whether gray rules and routing policies are configured accurately.
It can monitor the hierarchical tree relationship between gateway and service gray level.
Can monitor the whole link traffic topology

The author attempts to investigate a series of distributed tracking systems and middleware, including opentracking, Uber Jaeger, Twitter Zipkin, Apache Skywalking, Pinpoint, CAT, etc., and finally decides to use opentracking + Uber Jaeger to achieve it. The important reason is that in addition to ease of use and scalability, opentracking supports WebMvc and WebFlux, and the tracking system in the industry can support webfl. UX is relatively small

[opentracking] opentracking has entered CNCF and is providing a unified concept, specification, architecture and data standard for distributed tracking systems around the world. It provides platform independent and manufacturer independent API, which enables developers to easily add (or replace) the tracking system. For the call chain with multiple technology stacks coexisting, Opentracing adapts to Java, C, Go and. Net technology stacks to realize the full link distributed tracing function. To date, Uber Jaeger, Twitter Zipkin and Apache Skywalking have adapted to the Opentracing specification

The author takes the Discovery open source framework of Nepxion community as an example to integrate.

Source page, please visit https://github.com/Nepxion/Discovery

Guide home page, please visit https://github.com/Nepxion/DiscoveryGuide

Document home page, please visit https://pan.baidu.com/s/1i57rXaNKPuhGRqZ2MONZOA#list/path=%2FNepxion

Integrated renderings

Basic concepts

The gray call chain mainly includes the following 11 parameters. The user can define the call chain parameters to be passed, such as traceId, spanId, etc.; or the business call chain parameters to be passed, such as mobile, user, etc.

1. n-d-service-group - service group or application
 2. n-d-service-type - service type, divided into "gateway" and "service"
3. n-d-service-id - service ID
 4. n-d-service-address - service address, including Host and Port
 5. n-d-service-version - service version
 6. n-d-service-region - service region
 7. n-d-version - version route value
 8. n-d-region - region routing value
 9. n-d-address - address routing value
 10. n-d-version-weight - version weight routing value
 11. n-d-region-weight - region weight routing value

Core realization

Opentracing general module

Source reference https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-opentracing

Since the Spring Cloud Gateway, Zuul and service should be considered in opentracking extension, its core logic has certain encapsulation, so the author extracts a common module discovery plugin strategy opentracking, which includes configuration, operation, context and other modules, and focuses on the operation module. Other modules are relatively simple, not detailed one by one.

Before elaboration, the author needs to explain a configuration that will determine the core implementation and display of the terminal interface.

If it is on, the gray information is output to the independent Span node, which means that in the interface display, the gray information is displayed through the independent GRAY Span node. The advantage is that the information is concise, and the disadvantage is that the Span node will double. We can call it model A.
If it is turned off, the gray-scale information will be output to the native Span node, which means that in the interface display, the gray-scale information will be mixed with the call information and protocol information of the native Span node. The disadvantage is that the information is jumbled, and the advantage is that the number of Span nodes will not increase. We can call it pattern B.

# The gray information of the start and close call chain is output as an independent Span node in Opentracing. If it is closed, the gray information is output to the native Span node. Default to true if missing
spring.application.strategy.trace.opentracing.separate.span.enabled=true

Opentracing public operation class - strategyopentracing operation.java

Assembling Tracer objects injected into Opentracing
Opentraceninitialize method, which is provided for the initialization of Span nodes of gateways and services
- In mode A, tracer.buildSpan(...).start() implements the creation of A new Span and places it in the ThreadLocal of the strategy open tracing context of the storage context.
- [mode B], no work is required.
Opentracenheader method, which provides the gray call chain output to the gateway
- In mode A, first get the Span object from ThreadLocal of strategyopentraceingcontext, then put the elements of customationmap (custom call chain parameters) into Tag, finally put the main 11 parameters of gray call chain (obtained through strategyContextHolder.getHeader(...) and more context information into Tag.
- In mode B, similar to mode A, the only difference is the processing of Tags.COMPONENT. Since the native Span node already has this information, it does not need to be put into the Tag.
Opentracenglocal method, which provides the gray call chain output to the service
- In mode A, first get the Span object from the ThreadLocal of the strategyopentrainingcontext, then put all the elements of the customizationMap (custom call chain parameters) into the Tag, and finally put the main 11 parameters of the gray call chain (obtained through pluginAdapter.getXXX()) and more context information into the Tag.
- In mode B, similar to mode A, the only difference is the processing of Tags.COMPONENT. Since the native Span node already has this information, it does not need to be put into the Tag.
Opentracengerror method, which provides abnormal output of gray call chain to the service
- In mode A, first get the Span object from the ThreadLocal of the StrategyOpentracingContext, and then the span.log(...) method implements the exception output.
- [mode B], no work is required.
Opentracengclear method, Span reporting and clearing of gray call chain
- In mode A, first get the Span object from ThreadLocal of StrategyOpentracingContext, then the span.finish() method implements the Span report, and finally the StrategyOpentracingContext.clearCurrentContext() method implements the Span clear.
- [mode B], no work is required.
getCurrentSpan method
- In mode A, return StrategyOpentracingContext.getCurrentContext().getSpan(), which is the new Span object of opentracingInitialize.
- In mode B, tracer.activeSpan() is returned, which is the native Span object.

public class StrategyOpentracingOperation {
    private static final Logger LOG = LoggerFactory.getLogger(StrategyOpentracingOperation.class);

    @Autowired
    protected PluginAdapter pluginAdapter;

    @Autowired
    protected StrategyContextHolder strategyContextHolder;

    @Autowired
    private Tracer tracer;

    @Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_ENABLED + ":false}")
    protected Boolean traceOpentracingEnabled;

    @Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_SEPARATE_SPAN_ENABLED + ":true}")
    protected Boolean traceOpentracingSeparateSpanEnabled;

    public void opentracingInitialize() {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = tracer.buildSpan(DiscoveryConstant.SPAN_VALUE).start();
        StrategyOpentracingContext.getCurrentContext().setSpan(span);

        LOG.debug("Trace chain for Opentracing initialized...");
    }

    public void opentracingHeader(Map<String, String> customizationMap) {
        if (!traceOpentracingEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing header");

            return;
        }

        if (MapUtils.isNotEmpty(customizationMap)) {
            for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
                span.setTag(entry.getKey(), entry.getValue());
            }
        }

        if (traceOpentracingSeparateSpanEnabled) {
            span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
        }
        span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
        span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
        span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
        span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, strategyContextHolder.getHeader(DiscoveryConstant.N_D_SERVICE_GROUP));
        ...

        String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
        if (StringUtils.isNotEmpty(routeVersion)) {
            span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
        }
        ...

        LOG.debug("Trace chain information outputs to Opentracing...");
    }

    public void opentracingLocal(String className, String methodName, Map<String, String> customizationMap) {
        if (!traceOpentracingEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing local");

            return;
        }

        if (MapUtils.isNotEmpty(customizationMap)) {
            for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
                span.setTag(entry.getKey(), entry.getValue());
            }
        }

        if (traceOpentracingSeparateSpanEnabled) {
            span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
        }
        span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
        span.setTag(DiscoveryConstant.CLASS, className);
        span.setTag(DiscoveryConstant.METHOD, methodName);
        span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
        span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
        span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, pluginAdapter.getGroup());
        ...

        String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
        if (StringUtils.isNotEmpty(routeVersion)) {
            span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
        }
        ...

        LOG.debug("Trace chain information outputs to Opentracing...");
    }

    public void opentracingError(String className, String methodName, Throwable e) {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing error");

            return;
        }

        span.log(new ImmutableMap.Builder<String, Object>()
                .put(DiscoveryConstant.CLASS, className)
                .put(DiscoveryConstant.METHOD, methodName)
                .put(DiscoveryConstant.EVENT, Tags.ERROR.getKey())
                .put(DiscoveryConstant.ERROR_OBJECT, e)
                .build());

        LOG.debug("Trace chain error outputs to Opentracing...");
    }

    public void opentracingClear() {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            span.finish();
        } else {
            LOG.error("Span not found in context to opentracing clear");
        }
        StrategyOpentracingContext.clearCurrentContext();

        LOG.debug("Trace chain context of Opentracing cleared...");
    }

    public Span getCurrentSpan() {
        return traceOpentracingSeparateSpanEnabled ? StrategyOpentracingContext.getCurrentContext().getSpan() : tracer.activeSpan();
    }

    public String getTraceId() {
        if (!traceOpentracingEnabled) {
            return null;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            return span.context().toTraceId();
        }

        return null;
    }

    public String getSpanId() {
        if (!traceOpentracingEnabled) {
            return null;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            return span.context().toSpanId();
        }

        return null;
    }
}

Opentracing Service module

Source reference https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-starter-service-opentracing

Realize the extension of OpenTracing to services, including configuration, tracer and other modules, and focus on tracer module. Other modules are relatively simple, not detailed.

Service tracing class of Opentracing - defaultservicestrategyopentracing tracer.java

Inherit DefaultServiceStrategyTracer and inject strategyopentracing operation
In the trace method, execute opentraceinitialize to initialize the Span first, so that the subsequent logic can get traceId and spanId from the Span, and execute opentracenglocal to realize the gray-scale call chain output of the service.
Executing opentracing error in error method to realize abnormal output of gray call chain of service
Open tracing clear is executed in release method to realize Span reporting and clearing of gray call chain

public class DefaultServiceStrategyOpentracingTracer extends DefaultServiceStrategyTracer {
    @Autowired
    private StrategyOpentracingOperation strategyOpentracingOperation;

    @Override
    public void trace(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
        strategyOpentracingOperation.opentracingInitialize();

        super.trace(interceptor, invocation);

        strategyOpentracingOperation.opentracingLocal(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), getCustomizationMap());
    }

    @Override
    public void error(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation, Throwable e) {
        super.error(interceptor, invocation, e);

        strategyOpentracingOperation.opentracingError(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), e);
    }

    @Override
    public void release(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
        super.release(interceptor, invocation);

        strategyOpentracingOperation.opentracingClear();
    }

    @Override
    public String getTraceId() {
        return strategyOpentracingOperation.getTraceId();
    }

    @Override
    public String getSpanId() {
        return strategyOpentracingOperation.getSpanId();
    }
}

Opentracing Spring Cloud Gateway module

Source reference https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-starter-gateway-opentracing

The implementation of OpenTracing's extension to Spring Cloud Gateway is similar to the discovery plugin strategy starter service OpenTracing module, which will not be described in detail.

Opentracing Zuul module

Source reference https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-starter-zuul-opentracing

The implementation of OpenTracing's extension to Zuul is similar to the discovery plugin strategy starter service OpenTracing module, which will not be described in detail.

Instructions

Example reference https://github.com/Nepxion/DiscoveryGuide

Usage mode

The Opentracing output mode is illustrated by Uber Jaeger. The steps are very simple.

from https://pan.baidu.com/s/1i57rXaNKPuhGRqZ2MONZOA#list/path=%2FNepxion Get Jaeger-1.14.0.zip, and run jaeger.bat after decompression under Windows operating system. Please study the Mac and Lunix operating systems by yourself.
After the Postman call, access the http://localhost:16686 View gray call chain
GRAY call chain supports WebMvc and WebFlux, and is marked with GRAY.

Switch control

The opening and closing of Opentracing call chain function shall be controlled by the following switches:

# Start and close the call chain. Default to false if missing
spring.application.strategy.trace.enabled=true
# Start and close the Opentracing output of the call chain, which supports the configuration of version F or later. Other versions do not need this line configuration. Default to false if missing
spring.application.strategy.trace.opentracing.enabled=true
# The gray information of the start and close call chain is output as an independent Span node in Opentracing. If it is closed, the gray information is output to the native Span node. Default to true if missing
spring.application.strategy.trace.opentracing.separate.span.enabled=true

Optional Features

The creation of custom call chain context parameter (this class is not required), inheriting defaultstrategytracerdadapter

// Creating custom call chain context parameters
// For getTraceId and getSpanId methods, when call chain middleware such as Opentracing is introduced, it is determined by the call chain middleware, and the definition here will not work; when call chain middleware such as Opentracing is not introduced, the definition here is valid, and the following code indicates that it is obtained from Http Header and passed through the whole link.
// For the getCustomizationMap method, it indicates the customized business parameters output to the call chain, which can be output to the call chain middleware such as log and Opentracing at the same time. In the following code, it indicates that it is obtained from the Http Header and passed through the whole link.
public class MyStrategyTracerAdapter extends DefaultStrategyTracerAdapter {
    @Override
    public String getTraceId() {
        return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID) : StringUtils.EMPTY;
    }

    @Override
    public String getSpanId() {
        return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID) : StringUtils.EMPTY;
    }

    @Override
    public Map<String, String> getCustomizationMap() {
        return new ImmutableMap.Builder<String, String>()
                .put("mobile", StringUtils.isNotEmpty(strategyContextHolder.getHeader("mobile")) ? strategyContextHolder.getHeader("mobile") : StringUtils.EMPTY)
                .put("user", StringUtils.isNotEmpty(strategyContextHolder.getHeader("user")) ? strategyContextHolder.getHeader("user") : StringUtils.EMPTY)
                .build();
    }
}

In the configuration class, create the call chain class in @ Bean mode, and override the built-in call chain class of the framework.

@Bean
public StrategyTracerAdapter strategyTracerAdapter() {
    return new MyStrategyTracerAdapter();
}

Author of this article

Ren Haojun, more than 10 years of open source experience, Github ID: @ HaojunRen, founder of Nepxion open source community, Nacos Group Member, spring cloud Alibaba & Nacos & sentinel committer

Please contact me.

WeChat, public numbers and documents

This article is based on the platform of blog one article multiple sending OpenWrite Release!

Posted by julieb on Mon, 28 Oct 2019 22:50:44 -0700

Programmer Group