Akka-Cluster(0) - Some Ideas for Distributed Application Development

Keywords: Scala Database Netty Programming

When I first came into contact with akka-cluster, I had a dream to make full use of the free distribution and independent operation of actors to achieve some kind of distributed program. The computational tasks of this program can be subdivided artificially and then assigned to actors distributed on multiple servers. These servers are all in the same cluster environment. They are all nodes in akka-cluster. The number of nodes in akka-cluster only needs to be increased or decreased randomly according to the requirement of computing power through system configuration. Distributed programs running on the cluster can automatically adjust the distribution of actors on each node without modifying the software, rebalance the operation load of the program, and continue to run without any influence.

Some akka-cluster situations are also introduced in the previous akka series blogs. Recently, in the topic series "Programming Patterns in Cluster Environments (PICE)", we discussed how to integrate different types of database services in cluster environment through protobuf-gRPC. Because the database services in the cluster are connected by akka-stream, we use Flow to send the program and data together as stream elements to the corresponding database services for processing. Then an idea arises: when the database service receives a service request (assuming that data processing is a time-consuming and resource-consuming task), it can divide the tasks, then distribute these small tasks to multiple nodes in the cluster to calculate, and collect the results according to the calculation requirements. Then if the number of servers can be increased or decreased arbitrarily according to the number of users and the size of computing tasks, the computing requirements of any scale can be satisfied. The most important thing is that this cluster node size adjustment must be some kind of configuration, that is, by modifying the configuration file, but without modifying the software code. These requirements are precisely the special capabilities of akka-cluster. So I decided to open a series of akka-cluster topics to discuss the distributed software development model in cluster environment.

The following ways provided by akka-cluster are more in line with our requirements:

1. distributed pub/sub-Distributed Publish and Subscribe Mode

2. cluster-singleton-single actor pattern

3. cluster-load-balancing-cluster load balancing mode

4. cluster-sharding-Cluster Fragmentation Model

In the following blogs in this series, we will discuss the details of how they are used in specific programming in a pattern-by-pattern manner. But first, we discuss how to define akka-cluster nodes through configuration files to achieve cluster size adjustment.

The life cycle of a cluster node goes through the following stages:

Joining->Up，Leaving->Exiting，Exiting->Removed，Unreachable->Up，Unreachable->Down，Down->Removed

Next, we use actor s running on different cluster nodes to observe the status transition of each node through the status transition messages of cluster members in the subscription system.

class EventListener extends Actor with ActorLogging {
  import EventListner._

  val cluster = Cluster(context.system)

  override def preStart(): Unit = {
    cluster.subscribe(subscriber = self,initialStateMode = InitialStateAsEvents
    ,classOf[MemberEvent],classOf[UnreachableMember])
    super.preStart()
  }
  override def postStop(): Unit = {
    cluster.unsubscribe(self)
    super.postStop()
  }

  override def receive: Receive = {
    case MemberJoined(member) =>
      log.info("{} is JOINING...", member.address)
    case MemberUp(member) =>
      log.info("{} is UP!", member.address)
    case MemberWeaklyUp(member) =>
      log.info("{} is weakly UP!", member.address)
    case MemberLeft(member) =>
      log.info("{} is LEAVING...", member.address)
    case MemberExited(member) =>
      log.info("{} is EXITING...", member.address)
    case MemberRemoved(member, prevStatus) =>
      log.info("{} is REMOVED! from state {}", member.address, prevStatus)
    case UnreachableMember(member) =>
      log.info("{} is UNREACHABLE!", member.address)
    case ReachableMember(member) =>
      log.info("{} is REACHABLE!", member.address)
    case UnreachableDataCenter(datacenter) =>
      log.info("Data Center {} is UNREACHABLE!", datacenter)
    case ReachableDataCenter(datacenter) =>
      log.info("Data Center {} is REACHABLE!", datacenter)
    case Leave =>
      cluster.leave(cluster.selfAddress)
      log.info("{} is asked to leave cluster.",cluster.selfAddress)
    case Down =>
      cluster.down(cluster.selfAddress)
      log.info("{} is asked to shutdown cluster.",cluster.selfAddress)
  }

}

Leave and Down are custom message types:

object EventListner {
  trait Messages {}
  case object Leave extends Messages
  case object Down extends Messages
  def props = Props(new EventListener)
...
}

The basic configuration file of akka-cluster is as follows:

akka {
  actor {
    provider = "cluster"
  }
  remote {
    log-remote-lifecycle-events = off
    netty.tcp {
      hostname = "localhost"
      port = 2551
    }
  }
  cluster {
    seed-nodes = [
      "akka.tcp://ClusterSystem@localhost:2551"]
  }
}

In fact, hostname,port,seed-nodes can be configured in the program. If necessary, we just need to specify in the configuration file that this is a cluster mode program. Other parameters are defined in the program:

akka {
  actor {
    provider = "cluster"
  }
}

Then we can configure missing cluster parameters in the program:

object EventListner {
  trait Messages {}
  case object Leave extends Messages
  case object Down extends Messages
  def props = Props(new EventListener)

def create(host: String = "localhost", port: Int = 0, seednode: String = "") = {
    var config = ConfigFactory.parseString(s"akka.remote.netty.tcp.hostname=${host}")
                 .withFallback(ConfigFactory.parseString(s"akka.remote.netty.tcp.port=${port}"))
    if (seednode.length > 0) {
      val strConfig = "akka.cluster.seed-nodes=[\"" + seednode + "\"]"
      val configSeed = ConfigFactory.parseString(strConfig)
      config = config.withFallback(configSeed)
    }
    config = config.withFallback(ConfigFactory.load("akka-cluster-config"))
    val clusterSystem = ActorSystem(name="ClusterSystem",config=config)
    clusterSystem.actorOf(Props[EventListener])
  }

}

In the create function, ConfigFactory.parseString converts a string into a cluster configuration parameter, and multiple parameters can be supplemented by withFallback definitions.

The following is the EventListener test program:

import EventListner._
object EventDemo extends App {

  val listner1 = EventListner.create(port = 2551)  //seed node
  scala.io.StdIn.readLine()
  val listner2 = EventListner.create()    //port=0 random port
  scala.io.StdIn.readLine()
  val listner3 = EventListner.create()    //port=0 random port

  scala.io.StdIn.readLine()

  listner3 ! Leave
  scala.io.StdIn.readLine()

  listner2 ! Down
  scala.io.StdIn.readLine()

  listner1 ! Leave
  scala.io.StdIn.readLine()

}

The first one to run must be seednode, because each node needs to connect to seednode at startup. The following is the output of each stage:

[INFO] [10/22/2018 18:50:40.888] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:2551] - Started up successfully
[INFO] [10/22/2018 18:50:40.931] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:2551] - Node [akka.tcp://ClusterSystem@localhost:2551] is JOINING itself (with roles [dc-default]) and forming new cluster
[INFO] [10/22/2018 18:50:40.933] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:2551] - Cluster Node [akka.tcp://ClusterSystem@localhost:2551] dc [default] is the new leader
[INFO] [10/22/2018 18:50:40.943] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:2551] - Leader is moving node [akka.tcp://ClusterSystem@localhost:2551] to [Up]
[INFO] [10/22/2018 18:50:41.037] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://ClusterSystem@localhost:2551/user/$a] akka.tcp://ClusterSystem@localhost:2551 is UP!

[INFO] [10/22/2018 18:50:47.363] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:51679 is JOINING...
[INFO] [10/22/2018 18:50:47.930] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:2551] - Leader is moving node [akka.tcp://ClusterSystem@localhost:51679] to [Up]
[INFO] [10/22/2018 18:50:47.931] [ClusterSystem-akka.actor.default-dispatcher-2] [akka.tcp://ClusterSystem@localhost:2551/user/$a] akka.tcp://ClusterSystem@localhost:51679 is UP!
[INFO] [10/22/2018 18:50:48.109] [ClusterSystem-akka.actor.default-dispatcher-24] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:51679 is UP!

[INFO] [10/22/2018 18:50:53.765] [ClusterSystem-akka.actor.default-dispatcher-17] [akka.tcp://ClusterSystem@localhost:51681/user/$a] akka.tcp://ClusterSystem@localhost:51681 is JOINING...
[INFO] [10/22/2018 18:50:53.930] [ClusterSystem-akka.actor.default-dispatcher-22] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:51681 is JOINING...
[INFO] [10/22/2018 18:50:54.929] [ClusterSystem-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:2551] - Leader is moving node [akka.tcp://ClusterSystem@localhost:51681] to [Up]
[INFO] [10/22/2018 18:50:54.929] [ClusterSystem-akka.actor.default-dispatcher-21] [akka.tcp://ClusterSystem@localhost:2551/user/$a] akka.tcp://ClusterSystem@localhost:51681 is UP!

[INFO] [10/22/2018 18:52:00.806] [ClusterSystem-akka.actor.default-dispatcher-32] [akka.tcp://ClusterSystem@localhost:51681/user/$a] akka.tcp://ClusterSystem@localhost:51681 is asked to leave cluster.
[INFO] [10/22/2018 18:52:00.807] [ClusterSystem-akka.actor.default-dispatcher-28] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:51681] - Marked address [akka.tcp://ClusterSystem@localhost:51681] as [Leaving]
[INFO] [10/22/2018 18:52:00.808] [ClusterSystem-akka.actor.default-dispatcher-42] [akka.tcp://ClusterSystem@localhost:51681/user/$a] akka.tcp://ClusterSystem@localhost:51681 is LEAVING...
[INFO] [10/22/2018 18:52:00.809] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:51679 is asked to shutdown cluster.
[INFO] [10/22/2018 18:52:00.809] [ClusterSystem-akka.actor.default-dispatcher-16] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:51679] - Marking node [akka.tcp://ClusterSystem@localhost:51679] as [Down]
[INFO] [10/22/2018 18:52:00.810] [ClusterSystem-akka.actor.default-dispatcher-19] [akka.tcp://ClusterSystem@localhost:2551/user/$a] akka.tcp://ClusterSystem@localhost:51681 is LEAVING...
[INFO] [10/22/2018 18:52:00.933] [ClusterSystem-akka.actor.default-dispatcher-22] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:51681 is LEAVING...
[INFO] [10/22/2018 18:52:01.101] [ClusterSystem-akka.actor.default-dispatcher-24] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:51679] - Shutting down myself
[INFO] [10/22/2018 18:52:01.102] [ClusterSystem-akka.actor.default-dispatcher-24] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:51679] - Shutting down...
[INFO] [10/22/2018 18:52:01.104] [ClusterSystem-akka.actor.default-dispatcher-24] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:51679] - Successfully shut down
[INFO] [10/22/2018 18:52:01.110] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:2551 is REMOVED! from state Up
[INFO] [10/22/2018 18:52:01.110] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:51679 is REMOVED! from state Down
[INFO] [10/22/2018 18:52:01.111] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://ClusterSystem@localhost:51679/user/$a] akka.tcp://ClusterSystem@localhost:51681 is REMOVED! from state Leaving

[INFO] [10/22/2018 18:52:02.925] [ClusterSystem-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:2551] - Leader is moving node [akka.tcp://ClusterSystem@localhost:51681] to [Exiting]
[INFO] [10/22/2018 18:52:02.926] [ClusterSystem-akka.actor.default-dispatcher-19] [akka.tcp://ClusterSystem@localhost:2551/user/$a] akka.tcp://ClusterSystem@localhost:51681 is EXITING...
[INFO] [10/22/2018 18:52:02.927] [ClusterSystem-akka.actor.default-dispatcher-18] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:51681] - Exiting, starting coordinated shutdown
[INFO] [10/22/2018 18:52:02.927] [ClusterSystem-akka.actor.default-dispatcher-21] [akka.tcp://ClusterSystem@localhost:51681/user/$a] akka.tcp://ClusterSystem@localhost:51681 is EXITING...
[INFO] [10/22/2018 18:52:02.934] [ClusterSystem-akka.actor.default-dispatcher-41] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@localhost:51681] - Exiting completed

Posted by Micah D on Sat, 26 Jan 2019 23:45:14 -0800

Programmer Group

Akka-Cluster(0) - Some Ideas for Distributed Application Development

Hot Keywords