nsq is a simple and easy-to-use distributed message middleware originally developed by bitly company. It can be used for real-time message service in large-scale systems and can process hundreds of millions of messages every day.

It has the following characteristics:

Distributed. It provides a distributed, decentralized and single-point fault-free topology, stable message transmission and publishing guarantee, and can have high fault tolerance and high availability.

Easy to expand. It supports horizontal scaling, does not have a centralized message broker, and built-in discovery services make it very easy to add nodes to the cluster.

Easy operation and maintenance. It is very easy to configure and deploy, and has high flexibility.

Highly integrated. There are now official Golang, Python, and JavaScript clients. The community also has client libraries in various languages for easy access. Custom clients are also very easy.

1. First of all, go to the official documents to see the usage:

https://nsq.io/overview/quick...

Download the corresponding binary executable file and follow the above steps locally. Look at the background of nsqadmin as follows:

2. Building nsq in docker environment

Reference Official Information Creation: docker-compose.yml

version: '2' # High version support 3
services:
  nsqlookupd:
    image: nsqio/nsq
    command: /nsqlookupd
    ports:
      - "4160:4160" # tcp
      - "4161:4161" # http

  nsqd:
    image: nsqio/nsq
    # If the broadcast address is not filled in, the default is oshostname (or virtual machine name), so lookupd will not connect, so write IP directly.
    command: /nsqd --broadcast-address=10.236.92.208 --lookupd-tcp-address=nsqlookupd:4160
    depends_on:
      - nsqlookupd
    ports:
      - "4150:4150" # tcp
      - "4151:4151" # http

  nsqadmin:
    image: nsqio/nsq
    command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
    depends_on:
      - nsqlookupd
    ports:
      - "4171:4171" # http

Execute docker-compose up-d to generate the corresponding three containers:

nsqgo_nsqd_1

nsqgo_nsqlookupd_1

nsqgo_nsqadmin_1

3. Golang uses nsq

Create Producer: producer.go

package main

import (
  "fmt"
  "log"
  "time"

  "github.com/nsqio/go-nsq"
)

func main() {
  config := nsq.NewConfig()
  p, err := nsq.NewProducer("127.0.0.1:4150", config)

  if err != nil {
    log.Panic(err)
  }

  for i := 0; i < 1000; i++ {
    msg := fmt.Sprintf("num-%d", i)
    log.Println("Pub:" + msg)
    err = p.Publish("testTopic", []byte(msg))
    if err != nil {
      log.Panic(err)
    }
    time.Sleep(time.Second * 1)
  }

  p.Stop()
}

Write 1000 num-1-1000 in a loop and send it to the message queue through p.Publish, waiting for consumption.

Creating Consumers: consumer.go

package main

import (
  "log"
  "sync"

  "github.com/nsqio/go-nsq"
)

func main() {
  wg := &sync.WaitGroup{}
  wg.Add(1000)

  config := nsq.NewConfig()
  c, _ := nsq.NewConsumer("testTopic", "ch", config)
  c.AddHandler(nsq.HandlerFunc(func(message *nsq.Message) error {
    log.Printf("Got a message: %s", message.Body)
    wg.Done()
    return nil
  }))

  // 1. Direct nsqd
  // err := c.ConnectToNSQD("127.0.0.1:4150")

  // 2. Discovery through nsqlookupd services
  err := c.ConnectToNSQLookupd("127.0.0.1:4161")
  if err != nil {
    log.Panic(err)
  }
  wg.Wait()
}

There are two ways to connect to nsqd:

1. Direct nsqd, suitable for standalone version;

2. Through nsqlookupd service discovery, it is suitable for cluster version.

Output results:

go run producer.go

2019/08/18 20:29:51 Pub:num-0
2019/08/18 20:29:51 INF    1 (127.0.0.1:4150) connecting to nsqd
2019/08/18 20:29:52 Pub:num-1
2019/08/18 20:29:53 Pub:num-2
2019/08/18 20:29:54 Pub:num-3
2019/08/18 20:29:55 Pub:num-4
2019/08/18 20:29:56 Pub:num-5
2019/08/18 20:29:57 Pub:num-6
2019/08/18 20:29:58 Pub:num-7
2019/08/18 20:29:59 Pub:num-8
2019/08/18 20:30:00 Pub:num-9
2019/08/18 20:30:01 Pub:num-10

go run consumer.go

2019/08/18 20:30:08 INF    1 [testTopic/ch] querying nsqlookupd http://127.0.0.1:4161/lookup?topic=testTopic
2019/08/18 20:30:08 INF    1 [testTopic/ch] (10.236.92.208:4150) connecting to nsqd
2019/08/18 20:30:08 Got a message: num-0
2019/08/18 20:30:08 Got a message: num-1
2019/08/18 20:30:08 Got a message: num-2
2019/08/18 20:30:08 Got a message: num-3
2019/08/18 20:30:08 Got a message: num-4
2019/08/18 20:30:08 Got a message: num-5
2019/08/18 20:30:08 Got a message: num-6
2019/08/18 20:30:08 Got a message: num-7
2019/08/18 20:30:08 Got a message: num-8
2019/08/18 20:30:08 Got a message: num-9
2019/08/18 20:30:08 Got a message: num-10

github source download:

https://github.com/astraw99/n...

[Summary]

a. a single nsqd can have multiple topics, and each topic can have multiple channels. channel receives copies of all the topic messages to achieve multicast distribution, and each message on channel is evenly distributed to its subscribers to achieve load balancing;

b. nsq is specially designed for distributed and clustered applications. It has advantages in dealing with SPOF(single point of failure), high availability and eventual consistency.