Linker2 introduction

Linkerd consists of control plane and data plane:

• The control plane is a set of services running in the Kubernetes namespace (linker default). These services can complete the aggregation of telemetry data, provide user oriented API, and provide control data to the data plane agent. They jointly drive the data plane.
• The data plane uses a lightweight agent written by rule, which is installed in each pod of the service and becomes a part of the data plane. It receives all the access traffic of the pod, and configures iptables through initContainer to correctly forward the traffic and intercept all the outgoing traffic. Because it is an additional tool and intercepts all the incoming and outgoing traffic of the service, it does not need to change the code. , you can even add it to a running service.

Borrow the official picture:

proxy is developed by rust, and its internal asynchronous runtime adopts Tokio Framework, service components tower.

This paper focuses on the overall logic of the interaction between proxy and destination components, and analyzes the internal operation logic of proxy.

Process analysis

Initialization

After proxy starts:

1. app::init initialization configuration
2. app::Main::new create main logic.
3. Add a new task proxy parts:: build proxy task in main. Run util.

A series of initialization work will be carried out in proxyparts:: build ﹣ proxy ﹣ task. Here, only DST ﹣ SVC is concerned. The creation code is:

    dst_svc = svc::stack(connect::svc(keepalive))
                .push(tls::client::layer(local_identity.clone()))
                .push_timeout(config.control_connect_timeout)
                .push(control::client::layer())
                .push(control::resolve::layer(dns_resolver.clone()))
                .push(reconnect::layer({
                    let backoff = config.control_backoff.clone();
                    move |_| Ok(backoff.stream())
                }))
                .push(http_metrics::layer::<_, classify::Response>(
                    ctl_http_metrics.clone(),
                ))
                .push(proxy::grpc::req_body_as_payload::layer().per_make())
                .push(control::add_origin::layer())
                .push_buffer_pending(
                    config.destination_buffer_capacity,
                    config.control_dispatch_timeout,
                )
                .into_inner()
                .make(config.destination_addr.clone())

There are two references to DST ﹣ SVC. One is the creation of cate:: Resolve:: resolver; the other is the creation of ProfilesClient.

Resolver

1. API ﹐ resolve:: Resolve:: new (DST ﹐ SVC. Clone()) creates a resolver object
2. Call outbound::resolve to create the map:: endpoint:: resolve type object, and pass it as the parameter resolve to the outbound::spawn function to start the exit thread.

In outbound::spawn, resolve is used to create a load balancing control layer and for subsequent routing control:

let balancer_layer = svc::layers()
        .push_spawn_ready()
        .push(discover::Layer::new(
            DISCOVER_UPDATE_BUFFER_CAPACITY,
            resolve,
        ))
        .push(balance::layer(EWMA_DEFAULT_RTT, EWMA_DECAY));

In discover::Layer::layer:

let from_resolve = FromResolve::new(self.resolve.clone());
let make_discover = MakeEndpoint::new(make_endpoint, from_resolve);
Buffer::new(self.capacity, make_discover)

Profiles

1. In ProfilesClient::new, call api::client::Destination::new(dst_svc) create client side of grpc to exist in member variable service.
2. The profiles'client object is then used for the creation of inbound and outbound (omitting extraneous code):

    let dst_stack = svc::stack(...)...
        .push(profiles::router::layer(
            profile_suffixes,
            profiles_client,
            dst_route_stack,
        ))
        ...

Where profiles::router::layer creates a Layer object and assigns the profiles'client to the get'routes member. Then in the service method, it will be called to the Layer::layer method, where a MakeSvc object will be created, and the value of the get ﹣ routes member is profiles ﹣ client.

Function

When a new connection comes over, after the connection object is obtained from listen, it will be handed to the call of linked2:: proxy:: proxy:: Server:: server, and finally the linked2:: u proxy｛http:: Balance:: makesvc:: call and linked2:: u proxy｛http:: Profiles:: Router:: makesvc:: call methods.

balance

In linker2 ﹣ proxy ﹣ http:: Balance:: makesvc:: Call:

1. Call inner.call(target), where the inner is the result of the previous Buffer::new.
2. Generate a new linker2 ﹣ proxy ﹣ http:: Balance:: makesvc object as Future return

First look at inner.call. It will trigger the call methods of Buffer, MakeEndpoint, FromResolve and other structures through internal layer by layer calls. Finally, it will trigger the resolution.resolve (target) created at the beginning, and call the API "resolve:: Resolve:: call" internally.

In api_resolve::Resolve::call:

    fn call(&mut self, target: T) -> Self::Future {
        let path = target.to_string();
        trace!("resolve {:?}", path);
        self.service
            // GRPC request, get k8s endpoint
            .get(grpc::Request::new(api::GetDestination {
                path,
                scheme: self.scheme.clone(),
                context_token: self.context_token.clone(),
            }))
            .map(|rsp| {
                debug!(metadata = ?rsp.metadata());
                // Get the result stream
                Resolution {
                    inner: rsp.into_inner(),
                }
            })
    }

Put the returned Resolution into MakeSvc again, and then look at its poll:

    fn poll(&mut self) -> Poll<Self::Item, Self::Error> {
        // This poll will call:
        //    linkerd2_proxy_api_resolve::resolve::Resolution::poll
        //    linkerd2_proxy_discover::from_resolve::DiscoverFuture::poll
        //    linkerd2_proxy_discover::make_endpoint::DiscoverFuture::poll
        // Finally, obtain poll < change < socketaddr, endpoint > > 
        let discover = try_ready!(self.inner.poll());
        let instrument = PendingUntilFirstData::default();
        let loaded = PeakEwmaDiscover::new(discover, self.default_rtt, self.decay, instrument);
        let balance = Balance::new(loaded, self.rng.clone());
        Ok(Async::Ready(balance))
    }

Finally, return to service Balance.

When a specific request comes, balance:: poll ready will be judged first:

    fn poll_ready(&mut self) -> Poll<(), Self::Error> {
        // Get update < endpoint >
        // Remove the removed from self.ready'services
        // Add the unready service structure of Insert to self.unready'u services
        self.poll_discover()?;
        // For the UnreadyService, call its poll, and internally call the poll ready of svc to determine whether the endpoint is available.
        // When available, add it to self.ready'services
        self.poll_unready();
        
        loop {
            if let Some(index) = self.next_ready_index {
                // Find the corresponding endpoint, and return if available
                if let Ok(Async::Ready(())) = self.poll_ready_index_or_evict(index) {
                    return Ok(Async::Ready(()));
                }
            }
            // Select endpoint with low load
            self.next_ready_index = self.p2c_next_ready_index();
            if self.next_ready_index.is_none() {
                // 
                return Ok(Async::NotReady);
            }
        }
    }

When ready, call the request req:

    fn call(&mut self, request: Req) -> Self::Future {
        // Find the next available svc and remove it from the ready \
        let index = self.next_ready_index.take().expect("not ready");
        let (key, mut svc) = self
            .ready_services
            .swap_remove_index(index)
            .expect("invalid ready index");

        // Forward request
        let fut = svc.call(request);
        // Add to unready
        self.push_unready(key, svc);

        fut.map_err(Into::into)
    }

profiles

In linker2 proxy http:: profiles:: Router:: makesvc:: Call:

        // Initiate a stream to get route and dst_override updates for this
        // destination.
        let route_stream = match target.get_destination() {
            Some(ref dst) => {
                if self.suffixes.iter().any(|s| s.contains(dst.name())) {
                    debug!("fetching routes for {:?}", dst);
                    self.get_routes.get_routes(&dst)
                } else {
                    debug!("skipping route discovery for dst={:?}", dst);
                    None
                }
            }
            None => {
                debug!("no destination for routes");
                None
            }
        };

After several judgments, profilesclient:: get routes will be called and the result will be saved in route stream.

Enter get routes:

    fn get_routes(&self, dst: &NameAddr) -> Option<Self::Stream> {
        // Create channel
        let (tx, rx) = mpsc::channel(1);
        // This oneshot allows the daemon to be notified when the Self::Stream
        // is dropped.
        let (hangup_tx, hangup_rx) = oneshot::channel();
        // Create a Daemon object (Future task)
        let daemon = Daemon {
            tx,
            hangup: hangup_rx,
            dst: format!("{}", dst),
            state: State::Disconnected,
            service: self.service.clone(),
            backoff: self.backoff,
            context_token: self.context_token.clone(),
        };
        // Call Daemon::poll
        let spawn = DefaultExecutor::current().spawn(Box::new(daemon.map_err(|_| ())));
        // Outgoing channel receiver
        spawn.ok().map(|_| Rx {
            rx,
            _hangup: hangup_tx,
        })
    }

Let's go to Daemon::poll:

    fn poll(&mut self) -> Poll<Self::Item, Self::Error> {
        loop {
            // Traverse state member state
            self.state = match self.state {
                // Unconnected
                State::Disconnected => {
                    match self.service.poll_ready() {
                        Ok(Async::NotReady) => return Ok(Async::NotReady),
                        Ok(Async::Ready(())) => {}
                        Err(err) => {
                            error!(
                                "profile service unexpected error (dst = {}): {:?}",
                                self.dst, err,
                            );
                            return Ok(Async::Ready(()));
                        }
                    };
                    // Construct grpc request
                    let req = api::GetDestination {
                        scheme: "k8s".to_owned(),
                        path: self.dst.clone(),
                        context_token: self.context_token.clone(),
                    };
                    debug!("getting profile: {:?}", req);
                    // Get request task
                    let rspf = self.service.get_profile(grpc::Request::new(req));
                    State::Waiting(rspf)
                }
                // Get reply from request while requesting
                State::Waiting(ref mut f) => match f.poll() {
                    Ok(Async::NotReady) => return Ok(Async::NotReady),
                    // Normal recovery
                    Ok(Async::Ready(rsp)) => {
                        trace!("response received");
                        // Flow recovery
                        State::Streaming(rsp.into_inner())
                    }
                    Err(e) => {
                        warn!("error fetching profile for {}: {:?}", self.dst, e);
                        State::Backoff(Delay::new(clock::now() + self.backoff))
                    }
                },
                // Receive reply
                State::Streaming(ref mut s) => {
                    // Process reply flow
                    // Note here that parameter 1 is the reply flow of get profile request.
                    //   Parameter 2 is the previously created channel sender
                    match Self::proxy_stream(s, &mut self.tx, &mut self.hangup) {
                        Async::NotReady => return Ok(Async::NotReady),
                        Async::Ready(StreamState::SendLost) => return Ok(().into()),
                        Async::Ready(StreamState::RecvDone) => {
                            State::Backoff(Delay::new(clock::now() + self.backoff))
                        }
                    }
                }
                // Exception, end request
                State::Backoff(ref mut f) => match f.poll() {
                    Ok(Async::NotReady) => return Ok(Async::NotReady),
                    Err(_) | Ok(Async::Ready(())) => State::Disconnected,
                },
            };
        }
    }

Next, proxy stream:

    fn proxy_stream(
        rx: &mut grpc::Streaming<api::DestinationProfile, T::ResponseBody>,
        tx: &mut mpsc::Sender<profiles::Routes>,
        hangup: &mut oneshot::Receiver<Never>,
    ) -> Async<StreamState> {
        loop {
            // Whether the sender is ready
            match tx.poll_ready() {
                Ok(Async::NotReady) => return Async::NotReady,
                Ok(Async::Ready(())) => {}
                Err(_) => return StreamState::SendLost.into(),
            }

            // Get a piece of data from grpc stream
            match rx.poll() {
                Ok(Async::NotReady) => match hangup.poll() {
                    Ok(Async::Ready(never)) => match never {}, // unreachable!
                    Ok(Async::NotReady) => {
                        // We are now scheduled to be notified if the hangup tx
                        // is dropped.
                        return Async::NotReady;
                    }
                    Err(_) => {
                        // Hangup tx has been dropped.
                        debug!("profile stream cancelled");
                        return StreamState::SendLost.into();
                    }
                },
                Ok(Async::Ready(None)) => return StreamState::RecvDone.into(),
                // Get the profile structure correctly
                Ok(Async::Ready(Some(profile))) => {
                    debug!("profile received: {:?}", profile);
                    // Analytical data
                    let retry_budget = profile.retry_budget.and_then(convert_retry_budget);
                    let routes = profile
                        .routes
                        .into_iter()
                        .filter_map(move |orig| convert_route(orig, retry_budget.as_ref()))
                        .collect();
                    let dst_overrides = profile
                        .dst_overrides
                        .into_iter()
                        .filter_map(convert_dst_override)
                        .collect();
                    // Construct the profiles::Routes structure and push it to the sender
                    match tx.start_send(profiles::Routes {
                        routes,
                        dst_overrides,
                    }) {
                        Ok(AsyncSink::Ready) => {} // continue
                        Ok(AsyncSink::NotReady(_)) => {
                            info!("dropping profile update due to a full buffer");
                            // This must have been because another task stole
                            // our tx slot? It seems pretty unlikely, but possible?
                            return Async::NotReady;
                        }
                        Err(_) => {
                            return StreamState::SendLost.into();
                        }
                    }
                }
                Err(e) => {
                    warn!("profile stream failed: {:?}", e);
                    return StreamState::RecvDone.into();
                }
            }
        }
    }

Back to the MakeSvc::call method, the route stream created earlier will be used to create a linked2:: proxy:: http:: profiles:: Router:: service task object, and in its poll ready method, get the profiles::Routes from route:: stream through poll route stream and call Update routes to create a specific and available route rule linked2:: router. So far, the route rule has been established. Wait for the specific request to come over and then call linkerd2_router::call in call to make the routing judgment for the request.

Icon

profile

summary

Each processing logic of the tower framework adopted by proxy is one of the layers, which only needs to be stacked layer by layer during development. However, because of this, the interfaces between the layers are very similar, so you should be careful not to make mistakes. For the logic of destination, when the destination component of linker2 receives the grpc request from the proxy, whenever there is any change in the endpoint or service profile, it will immediately send it through the stream. After the proxy receives the request, it adjusts the load balancing policy according to the endpoint, adjusts the route according to the service profile, and then processes the actual request of the user service through them.

About the ServiceMesher community

The service mesher community was founded in April 2018 by a group of volunteers with the same values and ideas.

Community focus areas include: container, micro service, Service Mesh, Serverless, embrace open source and cloud native, and strive to promote the vigorous development of Service Mesh in China.

Community official website: https://www.servicemesher.com