Thrift for the First Experience of RPC Framework

Keywords: Java Python Spring Maven

Thrift for the First Experience of RPC Framework

Version description: thrfit 0.12.0

Module description:

  • thrift-demo-java-api: Generating Java api with thrift
  • thrift-demo-java-server: Java implements the Thrift server
  • thrift-demo-java-client: Implementing Thrift Client in Java
  • thrift-demo-py-api: Using thrift to generate Python api
  • thrift-demo-py-server:Python Implementing Thrift Server
  • thrift-demo-py-client: Python Implementing Thrift Client

1 Preface

The last article, "Dubbo of the initial experience of RPC framework", experienced Ali's open source RPC framework, the framework experience is fairly good, the industry also uses more. However, it only supports the Java language, not cross-language. Here we will experience a good performance, good evaluation, and support cross-language RPC framework thrfit. This article will use Java and Python to implement thrift server and client, and make cross-call.

2 Project preparation

2.1 thrfit installation

There are many ways to install thrift, such as brew install thrfit in mac environment or source code compilation. Also in linux environment, centos can use yum, ubantu can use apt, of course, unix environment can be installed by source code compilation. Thrfit also supports the installation of window s environment, download exe binary installation files on the official website for installation. Http://www.apache.org/dyn/close.cgi?Path=/thrift/0.12.0/thrift-0.12.0.exe. Of course, the official also suggested that we use docker to install thrift environment, the installation steps of thrfit are not detailed here.

2.2 Creating Projects

Let's first look at the overall directory structure.

learn-demo-thrift/
├── README.md
├── pom.xml
├── thrfit-demo-java-server
├── thrift
├── thrift-demo-java-api
├── thrift-demo-java-client
├── thrift-demo-py-api
├── thrift-demo-py-client
└── thrift-demo-py-server

The functions of several modules have been described at the beginning of the article. Another package is a thrift folder, which is used to store our. thrirft generation files and scripts.

2.2.1 Create a Maven project: learn-demo-thrift

Here you can create a Maven project directly using IDEA, specify groupId as learn.demo, artifactId as thrift, version as 1.0, and project name as learn-demo-thrift.

[External link picture transfer failure (img-MsgvI9c9-1566360312305) (https://raw.githubusercontent.com/shirukai/images/master/532984c696fa47013335e2d56c243f.jpg)]

2.2.2 Create folder thrift

In the project you just created, create a folder named thrift to save our. thrift generated files and scripts.

2.2.3 Create Maven sub-module: thrift-demo-java-api

Create Maven sub-module in the project, specify groupId as learn.demo, artifactId as thrift-demo-java-api, specify version as 1.0, module name as thrift-demo-java-api

2.2.4 Create Maven sub-module: thrift-demo-java-server

Create Maven sub-module in the project, specify groupId as learn.demo, artifactId as thrift-demo-java-server, specify version as 1.0, module name as: thrift-demo-java-server

2.2.5 Create Maven sub-module: thrift-demo-java-client

Create a Maven sub-module in the project, specify groupId as learn.demo, artifactId as thrift-demo-java-client, specify version as 1.0, and module name as thrift-demo-java-client.

2.2.6 Create Python sub-module: thrift-demo-py-api

Create a Python sub-module in the project, Python environment selection 2.7, module name: thrift-demo-py-api

2.2.7 Create Python sub-module: thrift-demo-py-server

Create a Python sub-module in the project, Python environment selection 2.7, module name: thrift-demo-py-server

2.2.8 Create a Python sub-module: thrift-demo-py-client

Create a Python sub-module in the project, Python environment selection 2.7, module name: thrift-demo-py-client

[External link picture transfer failure (img-8rIwP8tV-1566360312307)(https://raw.githubusercontent.com/shirukai/images/master/9cd73f82cf66b7228a942c54dfd4128e.jpg)]

The final project framework is created and completed.

3 Thrift API Generation

As mentioned above, thrirft supports cross-language. It supports cross-language because its services are generated from. thrift files. We can automatically generate code by defining a thrift service in a fixed format and then specifying a service language. Here we simply define a service and a data type, and generate API s for java and python respectively.

3.1 Definition. thrift file

Create a file named demo.thrift in the thrift directory under the project. The file consists of three parts:

  • Namespace: Used to describe the generated language and package paths, such as namespace java learn.demo.thrift.api
  • Strct: Used to define data struct ures corresponding to entity classes in Java
  • service: Used to define services, which includes abstract methods of definition

Following the demo.thrift file, we define two namespace s, java and python, and define a complex data structure, including an id of int type and a name of string type, as well as a list of <\ strings > type. And it defines a service which contains two methods.

namespace java learn.demo.thrift.api
namespace py thrift_demo.api

struct DemoInfo{
    1:i32 id,
    2:string name,
    3:list<string> tags
}

service DemoService{

    DemoInfo getDemoById(1:i32 id);

    void createDemo(1:DemoInfo demo)
}

3.2 Create Generation Scripts

Once the thrift file is defined, we can use the thrift command for code generation, such as

thrift --gen java -out ../thrift-demo-java-api/src/main/java demo.thrift

gen specifies the generated language

out Specifies the Generation Path

For convenience, we directly create a shell script named gen-code.sh to generate java and python code at one time, as follows:

#!/usr/bin/env bash
thrift --gen java -out ../thrift-demo-java-api/src/main/java demo.thrift

thrift --gen py -out ../thrift-demo-py-api demo.thrift

3.3 Generating Code

Executing the gen-code.sh script generates code under thrift-demo-java-api and thrirft-demo-py-api. As shown in the following figure, the generated Python code

[External link picture transfer failure (img-Q1EVr1qO-1566360312307) (https://raw.githubusercontent.com/shirukai/images/master/ebd59b411c15bca946 abddfb9c2a468a.jpg)]

The generated Java code is shown in the following figure

3.3.1 Adding dependencies to thrift-demo-java-api

After generating Java code, let's open the code to see

[External link picture transfer failure (img-8hz OceGR-1566360312308) (https://raw.githubusercontent.com/shirukai/images/master/26dfbb0267856 c5a70f1d9738b8073.jpg)]

We found that the code was red, because we did not introduce thrift dependencies, so we introduced related dependencies in the pom file.

        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.12.0</version>
        </dependency>

3.3.2 Release thrift-demo-py-api

We lack Maven dependencies above, and we also lack Python package dependencies here. We can use pip to install thrift packages.

pip install thrift

It's important to note that Maven modules can introduce dependencies directly into other modules, but Python can't be used directly in different modules. So here we wrap the generated Python code. Under the thrift-demo-py-api module, create a setup.py file for packaging and installation. The contents are as follows

# encoding: utf-8
from setuptools import setup, find_packages

setup(name="thrift_demo_py_api",
      version="1.0",
      description="The api of thrift demo.",
      author="shirukai",
      author_email="shirukai@hollysys.com",
      url="https://shirukai.github.io",

      packages=find_packages(),
      scripts=[]
      )

Publish this module as a package to the environment

python setup.py install

So we can refer to it directly in other modules.

>>> from thrift_demo.api import DemoService
>>>

4 Java Implements Server and Client

Above, we have generated Java code related to thrift service in thrift-demo-java-api. Here we use Java to implement both server and client, both of which are integrated with SpringBook.

5.1 Server: thrift-demo-java-server

The server mainly implements two aspects:

  • Implementing abstract interfaces
  • Implementing service exposure

Before that, we need to modify the module slightly, because it's a spring boot project, so here we specify the project parent as spring boot.

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.0.RELEASE</version>
    </parent>

Then, api dependencies and spring boot dependencies are introduced.

        <!-- demo api -->
        <dependency>
            <groupId>learn.demo</groupId>
            <artifactId>thrift-demo-java-api</artifactId>
            <version>1.0</version>
            <scope>compile</scope>
        </dependency>
        <!-- spring boot -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>

5.1.1 Implementing abstract interfaces

When we define demo.thrift in Section 3, we define two abstract methods in service, and thrift generates an interface class for me.

public class DemoService {

  public interface Iface {

    public DemoInfo getDemoById(int id) throws org.apache.thrift.TException;

    public void createDemo(DemoInfo demo) throws org.apache.thrift.TException;

  }
  //......
 }

So on the server side, we need to implement this interface first. Create a class named DemoServiceImpl under the learn.demo.thrift.server.service package. This class inherits the DemoService.Iface interface and implements the two methods. The contents are as follows

package learn.demo.thrift.server.service;

import learn.demo.thrift.api.DemoInfo;
import learn.demo.thrift.api.DemoService;
import org.apache.thrift.TException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;

import java.util.HashMap;
import java.util.Map;

/**
 * Created by shirukai on 2019-06-27 09:28
 * Demo Service
 */
@Service
public class DemoServiceImpl implements DemoService.Iface {
    private static final Logger log = LoggerFactory.getLogger(DemoServiceImpl.class);
    private static final Map<Integer, DemoInfo> demoCache = new HashMap<>(16);

    @Override
    public DemoInfo getDemoById(int id) throws TException {
        log.info("The client invoke method: getDemoById");
        if (demoCache.containsKey(id)) {
            return demoCache.get(id);
        }
        return null;
    }

    @Override
    public void createDemo(DemoInfo demo) throws TException {
        log.info("The client invoke method: createDemo");
        demoCache.put(demo.id, demo);
    }
}

5.1.2 Implementing Service Exposure

Here we need to implement a thrift service exposure, expose a port, so that the client can communicate. Before that, we used the springboot unified configuration file to specify the ports that need to be exposed.

Create an application.properties configuration file under resources, specifying that the thrift service exposes a port of 7911

thrift.server.name=thrift-demo-server
thrift.server.port=7911

Then, like a normal SpringBook application, create a startup class and create an Application class under learn.demo.thrift.server to start the SpringBook application.

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
  //......
}

The implementation of Thrift service can be divided into several steps:

  1. Creating Processors
  2. Setting up a listening port
  3. Construct service parameters: specify processor, transport mode, transport protocol
  4. Creating services
  5. Start up service

Here we use the @Configuration form to inject our Thrift service into SpringBook applications. Create a static inner class ThriftServer Configuration in Application

    @Configuration
    static class ThriftServerConfiguration {
        private static final Logger log = LoggerFactory.getLogger(ThriftServerConfiguration.class);
        @Value(("${thrift.server.port}"))
        private int serverPort;
        @Autowired
        private DemoService.Iface demoService;

        @PostConstruct
        public void startThriftServer() throws TTransportException {
            // Creating Processors
            TProcessor processor = new DemoService.Processor<>(demoService);

            // Listening port
            TNonblockingServerSocket socket = new TNonblockingServerSocket(serverPort);

            // Constructing service parameters
            TNonblockingServer.Args args = new TNonblockingServer.Args(socket);

            // Setting Processor
            args.processor(processor);
            // Setting up the transmission mode
            args.transportFactory(new TFastFramedTransport.Factory());
            // Setting up Transport Protocol
            args.protocolFactory(new TBinaryProtocol.Factory());

            // Creating services
            TServer server = new TNonblockingServer(args);
            log.info("The application is starting thrift server on address 0.0.0.0/0.0.0.0:{}",serverPort);
            // Start up service
            server.serve();
        }
    }

5.2 Client: thrift-demo-java-client

The client is also integrated with SpringBook

  • Implement client and inject Spring in the form of Bean
  • Implement REST interface to demonstrate remote calls

Before that, we still need to modify the module slightly, because it is a spring boot project, so we specify the project parent as spring boot.

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.0.RELEASE</version>
    </parent>

Then, api dependencies and spring boot dependencies are introduced.

        <!-- demo api -->
        <dependency>
            <groupId>learn.demo</groupId>
            <artifactId>thrift-demo-java-api</artifactId>
            <version>1.0</version>
            <scope>compile</scope>
        </dependency>
        <!-- spring boot -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>

5.2.1 Implement client and inject Spring in the form of Bean

We need to get the IP address and port number of the remote thrift service from the configuration file, so we need to create an application.properties file first.

thrift.server.name=thrift-demo-client
thrift.server.ip=127.0.0.1
thrift.server.port=8911

Also create SpringBoot application startup class under learn.demo.thrift.client

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
    // ......
 }

Next is the key point of client implementation, which is divided into the following steps:

  1. Create Socket Connections
  2. Setting up the transmission mode
  3. Setting up Transport Protocol
  4. Connect the server
  5. Get client instance

Similarly, in the form of @Configuration, client instances are injected into Spring in the form of beans. Create ThriftClientConfiguration static inner class under Application class

    @Configuration
    static class ThriftClientConfiguration {
        private static final Logger log = LoggerFactory.getLogger(ThriftClientConfiguration.class);
        @Value("${thrift.server.ip}")
        private String serverIp;
        @Value("${thrift.server.port}")
        private int serverPort;

        @Bean("demoService")
        public DemoService.Iface createThriftClient() throws TTransportException {
            // Create socket
            TTransport socket = new TSocket(serverIp, serverPort);
            // transmission mode
            TFramedTransport transport = new TFramedTransport(socket);
            // transport protocol
            TProtocol protocol = new TBinaryProtocol(transport);
            // Create a connection
            transport.open();
            log.info("The application is creating thrift client from address {}:{} ......",serverIp,serverPort);
            return new DemoService.Client(protocol);
        }
    }

5.2.2 Implement REST interface to demonstrate remote calls

This is the basic content of SpringBoot Web development. We have injected Thrift client instances into Spring in the form of beans. Here we can get the instance directly through @Autowire and call its method. Create DemoController class under learn.demo.thrift.client.controller to implement two REST interfaces.

package learn.demo.thrift.client.controller;

import learn.demo.thrift.client.dto.DemoInfoDTO;
import learn.demo.thrift.api.DemoInfo;
import learn.demo.thrift.api.DemoService;
import org.apache.thrift.TException;
import org.springframework.beans.BeanUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

/**
 * Created by shirukai on 2019-06-27 10:03
 * controller
 */
@RestController
@RequestMapping(value = "/demo")
public class DemoController {
    @Autowired
    private DemoService.Iface demoService;

    @PostMapping
    public String creatDemo(
            @RequestBody() DemoInfoDTO demoInfoDTO
    ) throws TException {
        DemoInfo demoInfo = new DemoInfo();
        BeanUtils.copyProperties(demoInfoDTO, demoInfo);
        demoService.createDemo(demoInfo);
        return "success";
    }

    @GetMapping(value = "/{id}")
    public DemoInfoDTO getDemo(
            @PathVariable("id") Integer id
    ) throws TException {
        DemoInfo demoInfo = demoService.getDemoById(id);
        if (demoInfo != null) {
            DemoInfoDTO demoInfoDTO = new DemoInfoDTO();
            BeanUtils.copyProperties(demoInfo, demoInfoDTO);
            return demoInfoDTO;
        }
        return null;
    }
}

5 Python Implements Server and Client

Above, we have generated Python code related to thrift service in thrift-demo-py-api, and packaged and published the module. Here we will use Python to implement the server and client.

5.1 Server: thrift-demo-py-server

Regardless of the language, the creation process of thrift's server and client is the same. There are not many articles here. The server is still implemented in two aspects:

  • Implementing abstract interfaces
  • Implementing service exposure

Python does not have the concept of interface, but Thrift still implements an abstract "interface" for the sake of unification, which is actually a class without concrete implementation. As follows:

class Iface(object):
    def getDemoById(self, id):
        """
        Parameters:
         - id

        """
        pass

    def createDemo(self, demo):
        """
        Parameters:
         - demo

        """
        pass

So first inherit this class and override its methods

class DemoServiceHandler(DemoService.Iface):
    """
    //Inherit DemoService.Iface and override its method
    """

    def getDemoById(self, id):
        print "The client invoke method: " + "getDemoById."
        if id in demoCache:
            return demoCache[id]
        else:
            return None

    def createDemo(self, demo):
        print "The client invoke method: " + "createDemo."
        demoCache[demo.id] = demo

Then there's the exposure of the server, the old routine.

  1. Creating Processors
  2. Setting up a listening port
  3. Initialization of transmission mode
  4. Initialization Transfer Protocol
  5. Creating services
  6. Start up service

Code directly

if __name__ == '__main__':
    handler = DemoServiceHandler()

    # Creating Processors
    processor = DemoService.Processor(handler)

    # Listening port
    transport = TSocket.TServerSocket("127.0.0.1", "8911")

    # Transfer Mode Factory: TBuffered Transport Factory/TFramed Transport Factory
    # What transmission mode does the server use, and what transmission mode does the client need to use?
    tfactory = TTransport.TFramedTransportFactory()

    # Transport Protocol Factory: TCompact Protocol/TJSONProtocol/TBinaryProtocol
    # What transport protocol does the server use, and what transport protocol does the client need to use?
    pfactory = TBinaryProtocol.TBinaryProtocolFactory()

    # Creating services: TSimpleServer/TForking Server/TThreadedServer/TThreadPool Server
    server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)

    print "python thrift server start"
    server.serve()

5.2 Client: thrift-demo-py-client

The Python client does not integrate complex services. It creates the client directly and makes remote calls.

# encoding: utf-8
"""
@author : shirukai
@date : 2019-06-26 21:02
thrift consumer
//Official website: http://thrift.apache.org/tutorial/py#client
"""
from thrift.protocol import TBinaryProtocol
from thrift.transport import TSocket, TTransport
from thrift_demo.api import DemoService
from thrift_demo.api.ttypes import DemoInfo

if __name__ == '__main__':
    # Establishing socket
    transport = TSocket.TSocket('127.0.0.1', 7911)

    # Transport mode, consistent with the server
    transport = TTransport.TFramedTransport(transport)

    # Transport protocol, consistent with the server
    protocol = TBinaryProtocol.TBinaryProtocol(transport)

    # Create Client
    client = DemoService.Client(protocol)

    # Connect the server
    transport.open()

    # Remote Call
    demo = DemoInfo()
    demo.id = 1
    demo.name = "demo1"
    demo.tags = ['1', '2']
    client.createDemo(demo)

    print client.getDemoById(1)

6 Cross-validation

So far, we have completed the development of Thrift server and client in Java version and the development of service and client in Python version. The following cross-validation will be carried out to verify whether the RPC we implemented is available through the following schemes.

  • Java Client-Java Server
  • Java Client-Python Server
  • Python Client-Python Server
  • Python Client-Java Server

6.1 Java Client-Java Server

First, we start the Java server and execute the main method of the Application in thrift-demo-java-server. Exposure of Thrift service port 7911

[Img-aA3aomew-1566360312309 (https://raw.githubusercontent.com/shirukai/images/master/a85ebfeee813eee937f2551caeb9b409.jpg)]

Before starting the Java client, you need to modify the configuration file and change the service port you will need to call to 7911

thrift.server.port=7911

Then start the application

[Img-taXqchlK-1566360312309 (https://raw.githubusercontent.com/shirukai/images/master/96f111535a46793360a51c01401dfc6c.jpg)]

When the startup is complete, we can make remote calls through REST.

Create Demo

[Img-aTZS80Xl-1566360312310 (https://raw.githubusercontent.com/shirukai/images/master/76b60b4e0c5111f092cb65ecb4fa80ac.jpg)]

The server prints the log indicating that the createDemo method has been invoked

2019-06-27 15:11:53.395  INFO 43138 --- [       Thread-2] l.d.t.server.service.DemoServiceImpl     : The client invoke method: createDemo

Get Demo

[Img-fkvy2aAo-1566360312310 (https://raw.githubusercontent.com/shirukai/images/master/89ae3280b5fce569d8be8475445a8e.jpg)]

The server prints the log and gets the corresponding, indicating that RPC is normal.

2019-06-27 15:13:27.957  INFO 43138 --- [       Thread-2] l.d.t.server.service.DemoServiceImpl     : The client invoke method: getDemoById

6.2 Java Client-Python Server

Now we stop the above two services, change the former Java server to Python server, and start the main method in the server under the thrift-demo-py-server module. Exposure service port 8911

[Img-RjKe353F-1566360312310 (https://raw.githubusercontent.com/shirukai/images/master/2c43e404f037d4fee7b4afaa18ba7549.jpg)]

Change the port of the server that needs to be invoked in the Java client to 8911

thrift.server.port=8911

Then start the service.

PostMan is still used for REST requests.

Create Demo

[Img-trsdO0Ge-1566360312313 (https://raw.githubusercontent.com/shirukai/images/master/89ae3280b5fce569d8be8475445a8e.jpg)]

Server Print Log

The client invoke method: createDemo.

Get Demo

[External link picture transfer failure (img-L07dSG85-1566360312314) (https://raw.githubusercontent.com/shirukai/images/master/89ae3280b5fce569d8be8475445a8e.jpg)]

After verification, it shows that our RPC is normal.

6.3 Python Client-Python Server

Similarly, Python client-Python server verification is carried out.

[Img-vBTnuizb-1566360312314 (https://raw.githubusercontent.com/shirukai/images/master/795beb70286227fe21a336b0bb0b4498.gif)]

6.4 Python Client-Java Server

Similar Python Client-Java Server Verification

[Img-0 EroMOXM-1566360312314 (https://raw.githubusercontent.com/shirukai/images/master/86e7ce8a4b1d614f5168325a860226fe.gif)]

7 Summary

Thrift's first experience so far, compared with Dubbo's feeling, did not walk too many pits. Because thrift has no registry and communicates directly, it is not complicated to configure.

Posted by dirkie on Tue, 20 Aug 2019 21:21:15 -0700

Hot Keywords

©2024 Programmer Group