I read a lot on the Internet and copied the official website README. I didn't explain many important things clearly. I had to study it myself.
My blog, keyword Less-Bug.com, welcome to pay attention.
Introduction to NSQ
- nsqd: daemon, client communication. Default port 4150 (TCP) 4151 (HTTP)
- Nsqlookupd: equivalent to a router. The client can discover producer and nsqd broadcast topics through it. An nsqlookupd can manage a group of nsqds. Default port: 4160 (TCP), 4161 (HTTP)
- nsqadmin: online panel, which can be accessed directly through the browser. Default port: 4171
Starting from the command line
Binary files can be downloaded directly. Open three terminals and execute them respectively:
nsqlookupd nsqd --lookupd-tcp-address=127.0.0.1:4160 --broadcast-address=127.0.0.1 nsqadmin --lookupd-http-address=127.0.0.1:4161
Use of go NSQ
I encapsulated a package:
package mq
import (
"encoding/json"
"fmt"
"time"
"github.com/nsqio/go-nsq"
"go.uber.org/zap"
)
type MessageQueueConfig struct {
NsqAddr string
NsqLookupdAddr string
SupportedTopics []string
}
type MessageQueue struct {
config MessageQueueConfig
producer *nsq.Producer
consumers map[string]*nsq.Consumer
}
func NewMessageQueue(config MessageQueueConfig) (mq *MessageQueue, err error) {
zap.L().Debug("New message queue")
producer, err := initProducer(config.NsqAddr)
if err != nil {
return nil, err
}
consumers := make(map[string]*nsq.Consumer)
for _, topic := range config.SupportedTopics {
nsq.Register(topic,"default")
consumers[topic], err = initConsumer(topic, "default", config.NsqAddr)
if err != nil {
return
}
}
return &MessageQueue{
config: config,
producer: producer,
consumers: consumers,
}, nil
}
func (mq *MessageQueue) Run() {
for name, c := range mq.consumers {
zap.L().Info("Run consumer for " + name)
// c.ConnectToNSQLookupd(mq.config.NsqLookupdAddr)
c.ConnectToNSQD(mq.config.NsqAddr)
}
}
func initProducer(addr string) (producer *nsq.Producer, err error) {
zap.L().Debug("initProducer to " + addr)
config := nsq.NewConfig()
producer, err = nsq.NewProducer(addr, config)
return
}
func initConsumer(topic string, channel string, address string) (c *nsq.Consumer, err error) {
zap.L().Debug("initConsumer to " + topic + "/" + channel)
config := nsq.NewConfig()
config.LookupdPollInterval = 15 * time.Second
c, err = nsq.NewConsumer(topic, channel, config)
return
}
func (mq *MessageQueue) Pub(name string, data interface{}) (err error) {
body, err := json.Marshal(data)
if err != nil {
return
}
zap.L().Info("Pub " + name + " to mq. data = " + string(body))
return mq.producer.Publish(name, body)
}
type Messagehandler func(v []byte)
func (mq *MessageQueue) Sub(name string, handler Messagehandler) (err error) {
zap.L().Info("Subscribe " + name)
v, ok := mq.consumers[name]
if !ok {
err = fmt.Errorf("No such topic: " + name)
return
}
v.AddHandler(nsq.HandlerFunc(func(message *nsq.Message) error {
handler(message.Body)
return nil
}))
return
}
Use example:
m, err := mq.NewMessageQueue(mq.MessageQueueConfig{
NsqAddr: "127.0.0.1:4150",
NsqLookupdAddr: "127.0.0.1:4161",
SupportedTopics: []string{"hello"},
})
if err != nil {
zap.L().Fatal("Message queue error: " + err.Error())
}
m.Sub("hello", func(resp []byte) {
zap.L().Info("S1 Got: " + string(resp))
})
m.Sub("hello", func(resp []byte) {
zap.L().Info("S2 Got: " + string(resp))
})
m.Run()
err = m.Pub("hello", "world")
if err != nil {
zap.L().Fatal("Message queue error: " + err.Error())
}
err = m.Pub("hello", "tom")
if err != nil {
zap.L().Fatal("Message queue error: " + err.Error())
}
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
<-sigChan
os.Exit(0);It is mainly for decoupling, so that if we change to a queue such as Kalfa, we can not touch the business code.
Output results:
2021-11-07T19:13:41.886+0800 DEBUG mq/mq.go:29 New message queue 2021-11-07T19:13:41.886+0800 DEBUG mq/mq.go:58 initProducer to 127.0.0.1:4150 2021-11-07T19:13:41.887+0800 DEBUG mq/mq.go:65 initConsumer to hello/default 2021-11-07T19:13:41.887+0800 INFO mq/mq.go:84 Subscribe hello 2021-11-07T19:13:41.887+0800 INFO mq/mq.go:84 Subscribe hello 2021-11-07T19:13:41.887+0800 INFO mq/mq.go:51 Run consumer for hello 2021/11/07 19:13:41 INF 2 [hello/default] (127.0.0.1:4150) connecting to nsqd 2021-11-07T19:13:41.887+0800 INFO mq/mq.go:77 Pub hello to mq. data = "world" 2021/11/07 19:13:41 INF 1 (127.0.0.1:4150) connecting to nsqd 2021-11-07T19:13:41.888+0800 INFO mq/mq.go:77 Pub hello to mq. data = "tom" 2021-11-07T19:13:41.888+0800 INFO buqi-admin-backend/main.go:60 S1 Got: "world" 2021-11-07T19:13:41.888+0800 INFO buqi-admin-backend/main.go:63 S2 Got: "tom"
From the output results, we can confirm the fact that different consumers who subscribe to the same topic and the same channel will be load balanced when messages flood in - each Handler will only receive one message.
Problems encountered
TOPIC_NOT_FOUND
There are two reasons.
One is case sensitivity. Topic names are case sensitive, so hello and hello are two different topics. When writing code, you should standardize the operation: extract constants and maintain a list of all topics.
Second, Topic is not created. After the first pub, the corresponding topic/channel can be created. It is recommended to write a script and call the / topic/create interface to create it at one time, otherwise the message will not be received until the subscription is retried the second time, resulting in unexpected delay.
Discovery client polling HTTP
This is because NsqLookupd itself is an intermediary that can manage a bunch of nsqd s with different IP addresses, so we can't always connect to only one nsq, so we need to poll to confirm which clients there are.
For small projects, you can bypass NsqLookupd:
// c.ConnectToNSQLookupd(mq.config.NsqLookupdAddr)
c.ConnectToNSQD(mq.config.NsqAddr)How to make multiple consumers consume the same topic?
Obviously, according to the mechanism of nsq, we need to let consumers of the same topic use different channels. One way is to randomize the channel, for example, using a recursive increment as the channel name.
The second method is to define the channel name according to the purpose.
The third method: it is said that addconcurrent handlers can be used, which has not been studied.
The fourth method: We mediate the Handler and use a consumer to consume, but manually send the message to a custom pipeline in the application layer and let the pipeline filter process the message. I guess it also avoids some critical zone problems.
Let's try the fourth method. (code published to) GIST , Github username (Pluveto)
Implement pipelined Handler
package mq
import (
"encoding/json"
"fmt"
"time"
"github.com/nsqio/go-nsq"
"go.uber.org/zap"
)
type MessageQueueConfig struct {
NsqAddr string
NsqLookupdAddr string
EnableLookupd bool
SupportedTopics []string
}
type MessageQueue struct {
subscribers map[string]Subscriber
config MessageQueueConfig
producer *nsq.Producer
}
type Messagehandler func(v []byte) bool
// The first node of LinkedHandlerNode is the head node, and the Handler must be nil
type LinkedHandlerNode struct {
Handler *Messagehandler
Index int
NextNode *LinkedHandlerNode
}
type Subscriber struct {
HandlerHeadNode *LinkedHandlerNode
Consumer *nsq.Consumer
Handler nsq.HandlerFunc
}
func createProducer(addr string) (producer *nsq.Producer, err error) {
zap.L().Debug("initProducer to " + addr)
config := nsq.NewConfig()
producer, err = nsq.NewProducer(addr, config)
return
}
func createConsumer(topic string, channel string, address string) (c *nsq.Consumer, err error) {
zap.L().Debug("initConsumer to " + topic + "/" + channel)
config := nsq.NewConfig()
config.LookupdPollInterval = 15 * time.Second
c, err = nsq.NewConsumer(topic, channel, config)
return
}
func NewMessageQueue(config MessageQueueConfig) (mq *MessageQueue, err error) {
zap.L().Debug("New message queue")
producer, err := createProducer(config.NsqAddr)
if err != nil {
return nil, err
}
subscribers := make(map[string]Subscriber)
for _, topic := range config.SupportedTopics {
nsq.Register(topic, "default")
consumer, err := createConsumer(topic, "default", config.NsqAddr)
if err != nil {
return nil, err
}
// The header node does not participate in actual use, so Index = -1
headNode := &LinkedHandlerNode{Index: -1}
hubHandler := nsq.HandlerFunc(func(message *nsq.Message) error {
// Call each Handler in a circular chain
curNode := headNode.NextNode
// Throw a warning when no user-defined Handler exists
if(nil == curNode){
return fmt.Errorf("No handler provided!")
}
for nil != curNode {
msg := message.Body
zap.S().Debugf("handler[%v] for %v is invoked", curNode.Index, topic)
stop := (*curNode.Handler)(msg)
if stop {
zap.S().Debugf("the message has stopped spreading ")
break
}
curNode = curNode.NextNode
}
return nil
})
consumer.AddHandler(hubHandler)
subscribers[topic] = Subscriber{
Consumer: consumer,
HandlerHeadNode: headNode,
}
}
return &MessageQueue{
config: config,
producer: producer,
subscribers: subscribers,
}, nil
}
func (mq *MessageQueue) Run() {
for name, s := range mq.subscribers {
zap.L().Info("Run consumer for " + name)
if mq.config.EnableLookupd {
s.Consumer.ConnectToNSQLookupd(mq.config.NsqLookupdAddr)
} else {
s.Consumer.ConnectToNSQD(mq.config.NsqAddr)
}
}
}
func (mq *MessageQueue) IsTopicSupported(topic string) bool {
for _, v := range mq.config.SupportedTopics {
if v == topic {
return true
}
}
return false
}
// Pub sends a message to the message queue
func (mq *MessageQueue) Pub(topic string, data interface{}) (err error) {
if !mq.IsTopicSupported(topic) {
err = fmt.Errorf("unsupported topic name: " + topic)
return
}
body, err := json.Marshal(data)
if err != nil {
return
}
zap.L().Info("Pub " + topic + " to mq. data = " + string(body))
return mq.producer.Publish(topic, body)
}
// Sub subscribes to a message from the message queue
func (mq *MessageQueue) Sub(topic string, handler Messagehandler) (err error) {
if !mq.IsTopicSupported(topic) {
err = fmt.Errorf("unsupported topic name: " + topic)
return
}
zap.L().Info("Subscribe " + topic)
subscriber, ok := mq.subscribers[topic]
if !ok {
err = fmt.Errorf("No such topic: " + topic)
return
}
// Arrive at the last valid linked list node
curNode := subscriber.HandlerHeadNode
for nil != curNode.NextNode {
curNode = curNode.NextNode
}
// Create node
curNode.NextNode = &LinkedHandlerNode{
Handler: &handler,
Index: 1 + curNode.Index,
NextNode: nil,
}
return
}
The idea here is to create a unique Handler for each consumer in advance, which will call each specific Handler in the linked list in turn. When a user subscribes to a Topic, the Handler provided by the user is added to the end of the linked list.
Use example:
m, err := mq.NewMessageQueue(mq.MessageQueueConfig{
NsqAddr: "127.0.0.1:4150",
NsqLookupdAddr: "127.0.0.1:4161",
SupportedTopics: []string{"hello"},
EnableLookupd: false,
})
if err != nil {
zap.L().Fatal("Message queue error: " + err.Error())
}
m.Sub("hello", func(resp []byte) bool {
zap.L().Info("S1 Got: " + string(resp))
return false
})
m.Sub("hello", func(resp []byte) bool {
zap.L().Info("S2 Got: " + string(resp))
return true
})
m.Sub("hello", func(resp []byte) bool {
zap.L().Info("S3 Got: " + string(resp))
return false
})
m.Run()
err = m.Pub("hello", "world")
if err != nil {
zap.L().Fatal("Message queue error: " + err.Error())
}
err = m.Pub("hello", "tom")
if err != nil {
zap.L().Fatal("Message queue error: " + err.Error())
}
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
<-sigChan
os.Exit(0)Output:
2021-11-07T20:30:38.448+0800 DEBUG mq/mq.go:40 New message queue 2021-11-07T20:30:38.448+0800 DEBUG mq/mq.go:89 initProducer to 127.0.0.1:4150 2021-11-07T20:30:38.448+0800 DEBUG mq/mq.go:96 initConsumer to hello/default 2021-11-07T20:30:38.448+0800 INFO mq/mq.go:113 Subscribe hello 2021-11-07T20:30:38.448+0800 INFO mq/mq.go:113 Subscribe hello 2021-11-07T20:30:38.448+0800 INFO mq/mq.go:113 Subscribe hello 2021-11-07T20:30:38.448+0800 INFO mq/mq.go:82 Run consumer for hello 2021/11/07 20:30:38 INF 2 [hello/default] (127.0.0.1:4150) connecting to nsqd 2021-11-07T20:30:38.454+0800 INFO mq/mq.go:108 Pub hello to mq. data = "world" 2021/11/07 20:30:38 INF 1 (127.0.0.1:4150) connecting to nsqd 2021-11-07T20:30:38.455+0800 INFO mq/mq.go:108 Pub hello to mq. data = "tom" 2021-11-07T20:30:38.455+0800 DEBUG mq/mq.go:57 handler[0] for hello is invoked 2021-11-07T20:30:38.455+0800 INFO buqi-admin-backend/main.go:60 S1 Got: "world" 2021-11-07T20:30:38.455+0800 DEBUG mq/mq.go:57 handler[1] for hello is invoked 2021-11-07T20:30:38.455+0800 INFO buqi-admin-backend/main.go:64 S2 Got: "world" 2021-11-07T20:30:38.455+0800 DEBUG mq/mq.go:60 the message has stopped spreading 2021-11-07T20:30:38.455+0800 DEBUG mq/mq.go:57 handler[0] for hello is invoked 2021-11-07T20:30:38.455+0800 INFO buqi-admin-backend/main.go:60 S1 Got: "tom" 2021-11-07T20:30:38.455+0800 DEBUG mq/mq.go:57 handler[1] for hello is invoked 2021-11-07T20:30:38.455+0800 INFO buqi-admin-backend/main.go:64 S2 Got: "tom" 2021-11-07T20:30:38.455+0800 DEBUG mq/mq.go:60 the message has stopped spreading ^C
You can see that when the Handler returns true, the message propagation can be blocked.