For link tracking, most of them refer to Google's dapper paper: https://bigbully.github.io/Dapper-translation/.

Through the summary of the paper, the core elements of span are traceId, name, spanId, parentSpanId, and others can be defined according to their own business needs.

The core principle of link tracing is to assign spanId to each service when invoking a service based on a global traceId, record the name of the operation, transfer spans with these attributes with requests when invoking RPC services, and minimize the impact of span data on the performance of the original service at key nodes. The method is passed to our own trace collection service, which collects the data and stores the data. The call chain is sorted out and displayed graphically through the relationship between traceId and spanId and parentSpanId. There are many modes in delivering span to the server, including: directly calling through RPC service, writing to local disk, reading disk data through another process, writing to remote server, caching transmission, sending messages and so on. They can be selected according to their own needs. In principle, they have as little impact on the service book as possible. Body performance.

This article only brings the process of client acquisition of span, and the process of trace server acquisition and analysis of display links is omitted here.

Github code address is also posted here. You can download and run it directly if you want to see the code: https://github.com/whiteBX/wrpc

First of all, let's look at my definition of span:

public class Span {
    /**
     * Global unique id
     */
    String traceId;
    /**
     * Operational Name -- Take Method Name here
     */
    String operationName;
    /**
     * Current spanId
     */
    String spanId;
    /**
     * Caller spanId
     */
    String parentSpanId;
    /**
     * appCode
     */
    String appCode;
    /**
     * Current machine ip
     */
    String localIp;
    /**
     * Target machine ip
     */
    String remoteIp;
    /**
     * Timestamp, used to record access time
     */
    long   timestamp;
}

Above are some span attributes that I defined. Of course, you can add some that you need to use, such as exception records, etc. But in principle, spans should be as small as possible here. If the definition is too large, it will affect the amount of data transmitted per request, which will affect our service performance.

Modify the dynamic proxy class in our comsumer to handle span-related content before launching a remote call:

public <T> T getBean(final Class<T> clazz, final String appCode) {
    return (T) Proxy.newProxyInstance(clazz.getClassLoader(), new Class[] { clazz }, new InvocationHandler() {
        public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
            // Get the server address
            String serverHost = getServer(appCode);
            Span span = SpanBuilder.buildNewSpan(SpanHolder.get(), method.getName(), serverHost, appCode);
            //// TODO: 2018/10/25 Startup Thread Initiates rpc Call Remote Link Tracking Service Recording Tracking Log Instead of Logging Here
            System.out.println("Link tracking, calling remote services:" + JSON.toJSONString(span));
            BaseRequestBO baseRequestBO = buildBaseBO(span, clazz.getName(), method, JSON.toJSONString(args[0]));
            return JSON.parseObject(call(serverHost, JSON.toJSONString(baseRequestBO)), method.getReturnType());
        }
    });
}

Here the annotated open new threads call rpc to transfer data, you can see that you need to modify, for example, write to disk through other processes read transmission performance is often higher than this way.

Here's a look at the SpanBuilder used in the above code:

public class SpanBuilder {

    /**
     * Construct span
     * @param parentSpan
     * @return
     * @throws UnknownHostException
     */
    public static Span buildNewSpan(Span parentSpan, String operationName, String serverIp, String appCode) throws UnknownHostException {
        Span span = new Span();
        span.setLocalIp(InetAddress.getLocalHost().getHostAddress());
        if (parentSpan == null) {
            span.setTraceId(ShortUUIDUtils.nextId());
            span.setParentSpanId("0");
        } else {
            span.setTraceId(parentSpan.getTraceId());
            span.setParentSpanId(parentSpan.getSpanId());
        }
        span.setTimestamp(System.currentTimeMillis());
        span.setOperationName(operationName);
        span.setRemoteIp(serverIp);
        span.setAppCode(appCode);
        span.setSpanId(ShortUUIDUtils.nextId());
        return span;
    }

    /**
     * Building a new appCpde's Span
     * @param span
     * @param appCode
     * @return
     */
    public static Span rebuildSpan(Span span, String appCode) {
        Span newSpan = copy(span);
        newSpan.setAppCode(appCode);
        return newSpan;
    }

    /**
     * Copy
     * @param source
     * @return
     */
    public static Span copy(Span source) {
        if (source == null) {
            return null;
        }
        Span span = new Span();
        span.setTraceId(source.getTraceId());
        span.setOperationName(source.getOperationName());
        span.setSpanId(source.getSpanId());
        span.setParentSpanId(source.getParentSpanId());
        span.setAppCode(source.getAppCode());
        span.setLocalIp(source.getLocalIp());
        span.setRemoteIp(source.getRemoteIp());
        span.setTimestamp(source.getTimestamp());
        return span;
    }
}

In fact, this is a simple construction span, which is mainly the generation of traceId and spanId. I use a short code generator here. There is no code to paste, you can go to the github upload code to see, or directly use uuid is also possible. Here is just a need to ensure that there is no repetition on the basis of as short as possible.

Next is the transformation of the receiving data processing method at the provider end.

public void channelRead(ChannelHandlerContext ctx, Object msg) {
    System.out.println("The server receives the request:" + msg);
    try {
        // Resolve the class name + method name + request parameter type (method signature)
        BaseRequestBO baseRequestBO = JSON.parseObject(msg.toString(), BaseRequestBO.class);
        // Put in span
        SpanHolder.put(baseRequestBO.getSpan());
        // Access to Registered Services
        Object object = ProviderBeanHolder.getBean(baseRequestBO.getClazzName());
        if (object == null) {
            System.out.println("Service class not registered:" + baseRequestBO.getClazzName());
        }
        // Invoking services through reflection
        Class paramType = Class.forName(baseRequestBO.getParamTypeName());
        Method method = object.getClass().getDeclaredMethod(baseRequestBO.getMethodName(), paramType);
        Object response = method.invoke(object, JSON.parseObject(baseRequestBO.getData(), paramType));
        // Request response
        ctx.writeAndFlush(JSON.toJSONString(response));
        Span span = SpanBuilder.rebuildSpan(baseRequestBO.getSpan(), ProviderConstant.APP_CODE);
        //// TODO: 2018/10/25 Startup Thread Initiates rpc Call Remote Link Tracking Service Recording Tracking Log Instead of Logging Here
        System.out.println("Link tracking, remote service response:" + JSON.toJSONString(span));
    } catch (Exception e) {
        System.out.println("Service exception" + e);
    }
}

Here we get the passed span and record it in the local thread variable. When we continue to call other provider s in the service processing, we can get this span in the comsumer code. When we regenerate a new span, the traceId of the span will be used, and the spanId will be set as the parentSpanId of the next span. A call chain is formed by the transfer at the sample level.

The main code here is actually completed, you can go directly to the github pull code to run. Here are some additional points:

1. Because span s are passed through local thread variables within services, links will be lost when new threads are in services. This can be solved by other ways, such as storing third-party caches.
2. span acquisition node, which uses two nodes to collect data before the consumer initiates the call and after the provider process is completed, is a scheme that considers the request success/timeout/exception comprehensively. You can also collect in other nodes, such as consumer receives response, provider receives request and so on, or collect them all, and then differentiate and process them on the trace server.