system function

summary

In order to avoid redundant code, the function of message push is taken as an independent platform. Each subsystem generates tasks through common RESTful interface or message (kafka).
Push function can be email, SMS, WebSocket or App. Tasks can be executed immediately or regularly. In order to meet the function of multiple reminders, cyclic tasks are extended on the basis of delayed tasks. Circular task is a special delay task. After the task is executed, dynamic reinjection tasks are marked according to the task type until all tasks are completed. (see system code for details)

Structure chart

Technical difficulties

redis task queue

1. Scheduled task | task queue

There are many ways to implement timing tasks. Node timing module node schedule and timer in native API are suitable for single task execution, not for multi task parallel (not suitable! ==No).

*Message queue is introduced to decouple services;
*As far as the system itself is concerned, push task may have the problems of large message volume and high concurrent volume, and the introduction of message queue can play a role of peak clipping
 *There is a particularly important point in message push: asynchrony. In most cases, the caller of the service may not have a high demand for callback of message push results. After the message producer initiates a consumption request through a specific channel, it can continue the subsequent process; (only suitable for the scenario that does not rely on execution feedback [serial not suitable])

The system uses the better module bull in node ecology, The flow chart is as follows: the system caller initiates the task request. The system first verifies the validity of the request (see the next section for permission verification), and then push es it into the task execution stack according to different types. The tasks in the Bull have roughly six states: wait, delay, activation, completion and failure. When the task state in the task stack is marked as activation, the task enters the execution stage Until the end of the next task.

* bull:github: https://github.com/OptimalBits/bull
 *bull: the publishing and subscription features of redis used at the bottom

Authority verification

1. Identity verification
In order to prevent the system from being called maliciously, it is necessary to do identity authentication at the level of malicious injection task, application portal and gateway. This article only explains the authentication of application portal, and the verification of gateway will be explained in the blog later.
In the system, the mainstream JWT token mechanism is also used, and the service generates token. After that, every request of the client must carry token. For the design of single microservice, the identity authentication of single system can be completely transferred to the gateway level.

2. Application authentication
Application authentication is the main function in a single system, in which crypto is the security module in node
See the following code block for a simple use of crypto

• crypto.publicEncrypt(key, buffer) (encryption)
• crypto.privateDecrypt(privateKey, buffer) (decryption)

import moment = require('crypto');
// encryption
const encrypt = (data, key) => {
    return crypto.publicEncrypt(key, Buffer.from(data));
}
// decrypt
const decrypt = (data, key) => {
    return crypto.privateDecrypt(key, encrypted);
}
const test = 'Test encrypted information'
// encryption
const crypted = encrypt(test, keys.pubKey); 
 // decrypt
const decrypted = decrypt(crypted, keys.privKey);

keys.privKey   keys.pubKey  Corresponding private key and public key
[crypto](https://nodejs.org/api/crypto.html)

System code

The specific implementation of each module is not explained in detail in this blog, but only the general method

• redis general method encapsulation

## redis encapsulation
import {HttpService, Inject, Injectable} from '@nestjs/common';
import { RedisService } from 'nestjs-redis';
@Injectable()
export class RedisCacheService {[https://github.com/OptimalBits/bull](https://github.com/OptimalBits/bull)
    public client;
    constructor(@Inject(RedisService) private redisService: RedisService) {
        this.getClient().then(r => {} );
    }
    async getClient() {
        this.client = await this.redisService.getClient();
    }

    // How to set values
    async set(key: string, value: any, seconds?: number) {
        value = JSON.stringify(value);
        if (!this.client) {
            await this.getClient();
        }
        if (!seconds) {
            await this.client.set(key, value);
        } else {
            await this.client.set(key, value, 'EX', seconds);
        }
    }

    // How to get the value
    async get(key: string) {
        if (!this.client) {
            await this.getClient();
        }
        const data: any = await this.client.get(key);
        if (!data) { return; }
        return JSON.parse(data);
    }
}

• bull task injection

// TaskService.ts
export class TaskService {
    constructor(
        // Inject the created task queue instance into the service
        @InjectQueue('email') private readonly emailNoticeQueue: Queue,
        @InjectQueue('message') private readonly messageNoticeQueue: Queue,
    ) { }
    // delayValue: delay time
    public async addTask() {
         await this.emailNoticeQueue.add('emitEmail', {
                 id: taskResult.insertedId,
                config: config.name,
         }, { delay: delayValue});
    }
}

• Task cycle processing function( email.processor.ts)

The bull cycle processing function can be referred to https://github.com/OptimalBit...

@Injectable()
@Processor('email')
export class NoticeEmailProcessor {
    constructor(
        @InjectQueue('email') private readonly emailNoticeQueue: Queue,
        private readonly redisCacheService: RedisCacheService,
        private readonly taskLogService: TaskLogService,
        private readonly taskEmitEmailService: TaskEmitEmailService,
        @Inject(TaskService)
        private readonly taskService: TaskService,
    ) {}

    @Process('emitEmail')
    public async handleTranscode(job: Job) {
     
    }

    /**
     * Next task ()
     */
    protected async nextLoopTak(task: TaskEntity, isSuccessFlag: boolean, status: number) {
       
    }
}

• Date processing adopted moment.js

https://github.com/moment/moment

System deployment

The system adopts docker deployment, and the system will start on port 10001 by default

1,DOCKERFILE

FROM ubuntu
MAINTAINER canyuegongzi
ENV HOST 0.0.0.0
RUN mkdir -p /app
COPY .. /app
WORKDIR /app
EXPOSE 10001
RUN apt-get update && \
    apt-get install redis-server && \
    npm config set registry https://registry.npm.taobao.org && \
    npm install && \
    npm install pm2 -g
CMD ["sh", "-c", "redis-server && pm2-runtime start ecosystem.config.js"]

2. pm2 startup script

## ecosystem.config.js
module.exports = [{
    script: 'dist/main.js',
    name: 'simpleNoticeCenterApi',
    exec_mode: 'cluster',
    instances: 2
}]

afterword

At present, the system is in the process of continuous iterative development, and there may be problems in different degrees. The common message push can be completed, and then the system security will be further improved, and other functions will be iterated in succession. The detailed code explanation will be explained in the subsequent blog, including the front and back ends.
Original address: http://blog.canyuegongzi.xyz/

resources

github

Message push system