title: Server Development Series 2
date: 2017-9-13 11:27:46

After three weeks of crazy overtime, the pace of server development can finally be put on hold, and there is time to take a good look at the project from an "external" perspective.

Write tcp server naked using swoole

In the absence of the Swiss Army Knife (familiar framework), naked swoole becomes the following:

• Basic skeleton of swoole tcp server

require_once __DIR__ . '/../vendor/autoload.php'; // composer autoload
require_once __DIR__ . '/config.php'; // configuration file

//---------------------server
$serv = new swoole_server("0.0.0.0", 9999);
$serv->set([
    'worker_num'            => 4,
    'task_worker_num'       => 8,
//    'daemonize'             => true,
    'pid'                   => __DIR__ . '/server.pid',
    'log_file'              => __DIR__ . '/../log/swoole.log',

    // Fixed Baotou Protocol
    'open_length_check'     => 1,       // Open Protocol Resolution
    'package_length_type'   => 'N',     // Types of Length Fields
    'package_length_offset' => 4,       // The N th byte is the value of the packet length
    'package_body_offset'   => 8,       // N th byte begins to calculate length
    'package_max_length'    => 2000000, // Maximum length of protocol
]);

// swoole table is the initial solution and can be used as global shared memory
//$swooleTable = new swoole_table('100000'); // up to 10w concurrent connection
//$swooleTable->column('auth', swoole_table::TYPE_INT, '1');
//$swooleTable->create();
//$serv->table = $swooleTable;

$serv->zoneId = $config['zone_id']; // Use configuration
$serv->userinfo = []; // Save information

// The redis/mysql connection must be created in the onWorkerStart callback
$serv->on('workerStart', 'onWorkerStart');

// mysql connection pool
$serv->on('task', 'onTask');
$serv->on('finish', 'onFinish');

$serv->on('connect', 'onConnect');
$serv->on('receive', 'onReceive'); // Message processing
$serv->on('close', 'onClose');

$serv->start();

• Message processing

Many function s are used here to separate logic

function onReceive(swoole_server $serv, $fd, $from_id, $data)
{
    //  The decode() function is used to parse the protocol, and autoload psr-4 is used to load it.
    $data = decode($data);
    // Each different message corresponds to a different function to process.
    if ($data['msg_type'] == 0) {
        function_msg0($serv, $fd, $data);
    } else if ($data['msg_type'] == 1) {
        function_msg1($serv, $fd, $data);
    }
}

• The redis/mysql connection must be created in the onWorkerStart callback

Refer to this wiki: Can I share a redis or mysql connection?
Here are two redis connections initialized at the start of each worker process, one for cache and one for pub/sub

function onWorkerStart(swoole_server $serv, $id){
    // Use only in worker processes
    if ($id < $serv->setting['worker_num']) {
        // cache
        $cache = new \Redis();
        $cache->connect($config['cache']['host'], $config['cache']['port'], $config['cache']['timeout']);
        $cache->auth($config['cache']['auth']);
        $serv->cache = $cache;

        // pub
        $pub = new \Redis();
        $pub->connect($config['pub_sub']['host'], $config['pub_sub']['port'], $config['pub_sub']['timeout']);
        $pub->auth($config['pub_sub']['auth']);
        $serv->pub = $pub;
    }
}

// This can then be used directly.
$serv->cache->set('key1', 'value1');
$serv->pub->publish('topic1', 'data1');

• mysql connection pool

Although the title is connection pool, there is only one mysql connection object instantiated here, and the principle is similar.

function onTask($serv, $task_id, $from_id, $data) {
    static $link = null;
    if ($link == null) {
        $link = mysqli_connect($config['mysql']['host'], $config['mysql']['user'], $config['mysql']['password'], $config['mysql']['database']);
        if (!$link) {
            $link = null;
            return;
        }
    }
    // Here we have made a layer of encapsulation, which needs to be marked on the original sql statements to determine what type of statements can be optimized.
    list($queryType, $sql) = explode('|', $data); 
    $result = $link->query($sql);
    if ($result) {
        if ($queryType == 'select') {
            $result = $result->fetch_all(MYSQLI_ASSOC);
        } else if ($queryType == 'insert') {
            $result = mysqli_insert_id($link);
        }
        return $result;
    }
}

function onFinish($serv, $data)
{
    //
}

// This can then be used directly.
$res = $serv->taskWait('select|select name from user where id=xxx'); // Here is synchronous blocking

Use the pit you stepped in when subscribing

For the implementation of subscription in swoole, you can refer to this blog: How to Forward Subscription Messages from Redis to WebSocket Client

Also start redis sub in the onWorkerStart callback function

function onWorkerStart(swoole_server $serv, $id){
//        if ($id == 0) {// / Start only one sub
            $sub = new swoole_redis(); // swoole_redis supports asynchrony
            $sub->on('message', function (swoole_redis $redis, $result) use ($serv, $config) {
                if ($result[0] == 'message') {
                    list($userId, $status) = explode(':', $result[2]);
                    // Resolve fd to send messages to client
                    $userFd = $serv->userinfo[$userId]['fd'] ?? 0;
                    if ($userId && $userFd && in_array($status, [0, 1, 2])) {
                        foreach ($serv->connections as $fd) {
                            if ($userFd == $fd) { // Only to the corresponding client

                                // Omitting business logic

                                // Send a message to client
                                $serv->send($fd, encode('foo'));
                                break;
                            }
                        }
                    }
                }
            });
            $sub->connect($config['pub_sub']['host'], $config['pub_sub']['port'], function (swoole_redis $redis, $result) use ($config) {
                $redis->auth($config['pub_sub']['auth'], function (swoole_redis $redis, $result) use ($config) {
                    $redis->subscribe('game_result_'. $config['zone_id']);
                });
            });
//        }
    }
}

Looking at the code carefully, you will find that such a line of comment "Start only one sub", which is determined by the business, when receiving subscription messages, only need to be forwarded to a specific user.
However, starting in the onWorkerStart callback function is not possible. The reason is that we need to understand swoole's process model first:

• When server starts, a master process starts first
• master process starts manager process and reactor thread
• reactor thread, used to manage tcp connection and tcp data sending and receiving
• The manager process is used to manage the worker process and task_worker process, according to the worker_num/task_worker_num configuration above.
• The worker process processes the data forwarded by the reactor thread. After processing the business logic, the data is sent to the reactor thread and forwarded to the user by the reactor thread.
• The worker process delivers the time-consuming task to the task_worker process, which triggers the onFinish event callback after processing.

So, we can judge whether we are in the worker process or task_worker process by using $work_id <$serv-> setting ['worker_num'].

When the first edition was written, I limited redis sub to $work_id = 0 only according to business requirements. However, the problem immediately arises: the current user's fd is not necessarily in the process of $work_id = 0, which will cause the following code to fail:

foreach ($serv->connections as $fd) { // The $serv actually corresponds to the current worker process
    if ($userFd == $fd) {
        // do something
    }
}

However, without the restriction of $work_id = 0, how many worker processes we have and how many sub-processes we have, leading to repeated subscriptions to messages and repeated business logic processing.

At this point, it's necessary to know what swoole provides. Process In the process management module, we only need to set up a single process to maintain the sub.

Problems found using swoole bare-write server

Obviously, the business logic above is not complex enough and the services used are not many, but the discomfort of the whole development is very obvious.

• Build development and test environments: compile swoole, install redis/mysql
• Configuration Management: Write to the business during rapid development, and extract to config.php configuration file when optimizing
• Service deployment: Start trying the official wiki system D daemonize solution, resulting in a large number of zombie processes
• Connection pooling: Connection pooling is necessary if higher performance is required, whether redis or mysql
• Protocol Processing: We use a fixed header + protobuf self-defined protocol, the protocol and business separation is a good design.
• Learning Cost: The first reading of the official wiki produces a general image, the second reading while implementing the examples in the wiki, and the third reading the corresponding chapters of the wiki according to business needs. But although the wiki is close to 1400 pages, there will still be new problems.

By contrast, there are so many web frameworks in php that MVC is popular. Is there a server framework in php that can solve these common problems?

Here's a recommendation swoole distribution When refactoring, this framework is chosen, and the advantages are briefly stated as follows:

• Docker configures the development environment, but docker for window mounting through directory will result in no hot update (there are solutions, of course).
• Serviceable deployment without relying on other services (system D supervisor)
• Pack Module Solves Protocol Resolution
• The classical MVC architecture can write business logic in controller s and model s with only minor modifications to the route.
• With connection pool, you can modify the configuration file.
• Yeah, there's also an itinerary.

$value = yield $this->redis_pool->getCoroutine()->get('key1');

• Yes, and Process

namespace app\Process;

use Server\Components\Process\Process;

class MyProcess extends Process
{
    public function start($process)
    {
        parent::start($process);

        // You can put the logic of redis sub here.
    }

    // This method can be invoked in controller using rpc
    public function getData()
    {
        return '123';
    }
}

Written in the end

Indeed, I haven't written about servers before. I have been staying at the stage of "talking on paper". When I really write, I find that "it's really tiring".
However, the books you read, the technical blog s you brush, and the technical conferences you attend in those years are always useful. What is blocked at the entry point is not the language, but the "foundation" of the field. These can be obtained through these ways, but the need to systematize them by yourself.
Of course, then coding, practice makes perfect will always be useful for a threaded programmer.