I've seen NSQ before and haven't seen it yet.Just when I have time today, I set up a simple try to get the message queue NSQ in this Go Language, which I'll briefly record here.

In fact, NSQ is used relatively little in China. I think I know such a thing here. Just look at the source code a little and learn something.

Introduction to NSQ

Several components of NSQ

1. nsqd: A daemon responsible for receiving, queuing, and forwarding messages to clients
2. nsqlookupd: A daemon for managing topology information, collecting topic and channel reported by nsqd, and providing eventually consistent discovery services
3. nsqadmin: A set of Web user interfaces for real-time viewing of cluster statistics and performing corresponding management tasks
4. utilities: Basic functionality, data flow processing tools such as nsq_stat, nsq_tail, nsq_to_file, nsq_to_http, nsq_to_nsq, to_nsq

Relevant web addresses

NSQ website: NSQ Official Website

GitHub: Github

Official NSQ Client: Official NSQ Client

NSQ Documentation: NSQ Document

NSQ protocol: NSQ Protocol

NSQ Installation

NSQ can be installed in several ways, such as binary, source, Docker, Brew, etc.

Binary Installation, available to Installation Address Download the Release package for your platform above and unzip it.

If it's a Mac computer, install it directly with Brew

brew install nsq

If it's a Docker installation, refer to the web address above. Just follow the steps. I don't use this installation method.

I'm using the source installation method, because the binary one is placed on top of S3 and slows down, so I download Github's source code directly. Another advantage here is that I can see the source code to track my learning.It's more convenient.

The downloaded directory structure is as follows:

NSQ Run

If you run from source instead of putting the executable in the bin directory after Make, after downloading all dependent packages and cd enters the nsqio/nsq/apps/nsqd directory, you can execute go run. /or go run main.go options.go otherwise the following error will be reported

nsqio/nsq/apps/nsqd/main.go:44:13: undefined: nsqdFlagSet
nsqio/nsq/apps/nsqd/main.go:54:10: undefined: config

Actually, if you enter the apps directory to execute, you will eventually execute the business processing code under nsqio/nsq/nsqd. Apps just wrap up the service with go-srv package, become a daemon and some entry parameters.

$ go run ./
[nsqd] 2020/03/22 00:55:27.597911 INFO: nsqd v1.2.1-alpha (built w/go1.11.2)
[nsqd] 2020/03/22 00:55:27.597980 INFO: ID: 809
[nsqd] 2020/03/22 00:55:27.598396 INFO: TOPIC(test): created
[nsqd] 2020/03/22 00:55:27.598449 INFO: TOPIC(test): new channel(test)
[nsqd] 2020/03/22 00:55:27.598535 INFO: TOPIC(test): new channel(lc)
[nsqd] 2020/03/22 00:55:27.598545 INFO: NSQ: persisting topic/channel metadata to nsqd.dat
[nsqd] 2020/03/22 00:55:27.599714 INFO: TCP: listening on [::]:4150
[nsqd] 2020/03/22 00:55:27.599806 INFO: HTTP: listening on [::]:4151

Seeing the hint above indicates successful startup, it opens TCP and HTTP ports, 4150,4151 can be changed by configuring or flag parameters, and it also supports TLS/SSL.

HTTP Test

After starting nsqd, you can use http to test sending a message and CURL to operate on it.

$ curl -d 'This is a test message' 'http://127.0.0.1:4151/pub?topic=test&channel=lc'
OK

NSQ message mode

We know that messages are generally pushed and pulled, and messages in NSQ are pushed, which ensures the timeliness of the message and pushes it out when there is a message.However, to control the consumability and rhythm of the root client, NSQ is achieved by changing the value of RDY.When there is no message, 0. After the server pushes the message, the client, for example, calls updateRDY(), changes this method to 3. When the server pushes, the value will flow.

Messages are sent over connected TCP, first under Topic, then under Channel, and finally msg is taken out of channel memoryMsgChan and sent out.

github.com/nsqio/nsq/nsqd/protocol_v2.go

func (p *protocolV2) messagePump(client *clientV2, startedChan chan bool) {
	var err error
	var memoryMsgChan chan *Message
	var backendMsgChan <-chan []byte
	var subChannel *Channel
	// NOTE: `flusherChan` is used to bound message latency for
	// the pathological case of a channel on a low volume topic
	// with >1 clients having >1 RDY counts
	var flusherChan <-chan time.Time
	var sampleRate int32

	subEventChan := client.SubEventChan
	identifyEventChan := client.IdentifyEventChan
	outputBufferTicker := time.NewTicker(client.OutputBufferTimeout)
	heartbeatTicker := time.NewTicker(client.HeartbeatInterval)
	heartbeatChan := heartbeatTicker.C
	msgTimeout := client.MsgTimeout
	
	...
    ...
	case msg := <-memoryMsgChan:
			if sampleRate > 0 && rand.Int31n(100) > sampleRate {
				continue
			}
			msg.Attempts++

			subChannel.StartInFlightTimeout(msg, client.ID, msgTimeout)
			client.SendingMessage()
			err = p.SendMessage(client, msg)
			if err != nil {
				goto exit
			}
			flushed = false
		case <-client.ExitChan:
			goto exit
		}

NSQ also supports delayed message delivery, such as when an order is not paid for invalid processing in 30 minutes, using the heap package's priority queue for delayed delivery, which implements some of the methods.By comparing the current time with the delay time, and then popping up the message from the delay queue and sending it to the channel, the follow-up process is the same as ordinary messages. I see that someone on the Internet has encountered that delay messages have concurrency problems. Finally, Redis ZSET is used to implement the delay, so I am not sure if the delay is reliable, the less demanding ones can be tried.

curl -d 'This is a delay message' 'http://127.0.0.1:4151/pub?topic=test&channel=lc&defer=3000'

defer Parameters in milliseconds

NSQ Consumption

When consuming messages, channels are similar to the concept of consumer groups within kafka, such as the same channel.Then only one instance will consume it, not many instances will consume that message, so it can be used for message load balancing. I see some people on the network are confused that they specify a topic and then consume it with different channels. They say how to receive messages from other channels, but they can't filter messages directly. Channels are not actually used for filtering.

The message sent by NSQ can be guaranteed to be consumed by at least one consumer, and its consumption level is at least once. In order to ensure message consumption, repeated consumption is unavoidable if the client times out, re-queues or reconnects, so the client business process must process the message equally.

Client reply FIN or REQ indicates success or resend.If the client fails to send the message in time, NSQ will send the message to the client repeatedly.

In addition, unlike Kafka, NSQ is able to get ordered messages, but NSQ is not, and the consumption received by clients is out of order.Although each message has a time stamp, it is important to note that there is a requirement for order.Therefore, NSQ is better suited for messages with large amounts of data but no order relationship between them.

Go Client for NSQ

NSQ supports many forms of clients, such as HTTP or client libraries, and it is officially recommended that you use HTTP, HTTP, to send GET or POST requests directly.

If you Go here, you can use the go-nsq library at go-nsq: go-nsq

go get https://github.com/nsqio/go-nsq

send message

package main

import (
	"errors"
	"fmt"
	"github.com/nsqio/go-nsq"
	"time"
)




func main() {
	var (
		p1 *producer
		p2 *producer
	)
	p1 = &producer{}
	p2 = &producer{}
	p1.producer,_ = InitProducer("127.0.0.1:4150")
	p2.producer,_ = InitProducer("127.0.0.1:4150")

	defer p1.producer.Stop()
	defer p2.producer.Stop()

	//p1.publish("test","hello!!!")
	p1.deferredPublish("test", 10 * time.Second,"This is a delay message?")
	fmt.Println("done")
}


type producer struct {
	producer *nsq.Producer
}

func(p *producer) publish(topic string,message string)  (err error){
	if message == "" {
		return errors.New("message is empty")
	}
	if err = p.producer.Publish(topic,[]byte(message)); err != nil {
		fmt.Println(err)
		return  err
	}
	return nil
}
// Delay message
func(p *producer) deferredPublish(topic string,delay time.Duration, message string)  (err error){
	if message == "" {
		return errors.New("message is empty")
	}
	if err = p.producer.DeferredPublish(topic,delay, []byte(message)); err != nil {
		fmt.Println(err)
		return  err
	}
	return nil
}

func InitProducer(addr string) (p *nsq.Producer,err error){
	var (
		config *nsq.Config
	)
	config = nsq.NewConfig()
	if p, err  = nsq.NewProducer(addr, config); err != nil {
		return  nil, err
	}
	return p, nil
}

Consumer News

package main

import (
	"encoding/json"
	"fmt"
	"github.com/nsqio/go-nsq"
)

//nsqio Consumption Test

type MyTestHandler struct {
	q *nsq.Consumer
	messageReceive int
}

func (h *MyTestHandler) HandleMessage(message *nsq.Message) error {
	type Data struct {
	}

	var (
		data *Data
		err error
	)
	data = &Data{}
	if err = json.Unmarshal(message.Body, data) ;err != nil {
		fmt.Printf("Id:%s, Msg:%s \n", message.ID, string(message.Body))
		err = nil
	}
	message.Finish()
	return nil
}
func initConsuemr(topic string, channel string) {
	var (
		config *nsq.Config
		h *MyTestHandler
		err  error
	)
	h = &MyTestHandler{
	}

	config = nsq.NewConfig()

	if h.q, err = nsq.NewConsumer(topic, channel, config); err != nil {
		fmt.Println(err)
		return
	}

	h.q.AddHandler(h)
	if err = h.q.ConnectToNSQD("127.0.0.1:4150"); err != nil {
		fmt.Println(err)
	}

	//<-h.q.StopChan
	fmt.Println("stop")
	return

}

func main() {
	initConsuemr("test","test")
	initConsuemr("test","lc")
	select{}
}

Overall, NSQ consumption is guaranteed and message reliability is guaranteed.Multiple nsqd and nsqlookupd can be used for distributed clustering, etc. channel with Go can be used for high concurrent consumption, high throughput and simple deployment.
However, I still don't feel as good as professional message queues like Kafka and RocketMQ, but they are sufficient in some scenarios.This has to be based on its own situation to choose between, after all, there is no good architecture, only the appropriate one.