Introduction to Akka Concurrent Programming Framework

Introduction to Akka

Akka is a toolkit for building highly concurrent, distributed, and scalable event-driven applications.Akka is a library developed using scala and can be used to develop Akka-based applications using scala and Java.

Akka characteristics

• Provides an asynchronous, non-blocking, high-performance event-driven programming model
• Built-in fault tolerance to allow Actor to recover or reset in case of an error
• Super lightweight event handling (millions of Actor s per GB heap)
• Using Akka, you can build high concurrent programs on a single machine or distributed programs on a network.

Akka communication process

The following pictures illustrate the basic flow of Akka Actor's concurrent programming model:

1. Students Create an ActorSystem
2. Create an ActorRef (teacher's reference) through ActorSystem and send a message to ActorRef
3. ActorRef sends messages to Message Dispatcher
4. Message Dispatcher saves messages in order to the MailBox of the target Actor
5. Message Dispatcher places MailBox in a thread
6. MailBox takes out the message in order and eventually passes it to a method accepted by TeacherActor

[External chain picture transfer failed (img-tV7CBK6r-1569035685343)(assets/1552871108166.png)]

Create Actor

Akka is also based on Actor for programming.Similar to Actor you've learned before.But Akka's ctor was written, created, and used to be somewhat different.

Introduction to API

ActorSystem

In Akka, ActorSystem is a heavyweight structure that requires multiple threads to be allocated, so in practice, ActorSystem is usually a single object that can be used to create many Actors.It is responsible for creating and supervising actors

Get ActorSystem in Actor

A reference to the Actor System that manages the Actor can be obtained by using context.system directly

Implement Actor Class

• Inherit Actor (Note: Import Actor under akka.actor package)
• Implement the receive method, which processes messages directly without adding loop and react method calls.Akka automatically calls receive to receive messages
• [Optional] You can also implement the preStart() method, which is executed after the Actor object is built and only once during the Actor declaration cycle

Load Akka Actor

1. To create an Actor for Akka, you must first get to create an ActorSystem.You need to give ActorSystem a name, and you can load some configuration items (which you'll use later)
2. Call ActorSystem.actorOf(Props(Actor object), "Actor name") to load Actor

Actor Path

Each Actor has a Path, which can be externally referenced just like writing a Controller/Handler using Spring MVC.The format of the path is as follows:

Getting Started Cases

Case Description

Create two Actor s based on Akka that send messages to each other.

[External chain picture transfer failed (img-ogFXt5Fp-1569035698456)(assets/1552879431645.png)]

Implementation Steps

1. Create Maven Module
2. Create and load Actor s
3. Send/Receive Messages

1. Create Maven modules

Using Akka requires importing the Akka library, so here we use Maven to manage the project

1. Create Maven Module
2. Open the pom.xml file to import akka Maven dependencies and plug-ins

2. Create and load Actor s

Create two Actor s

• SenderActor: used to send messages
• ReceiveActor: used to receive and reply to messages

Create Actor

1. Create ActorSystem
2. Create a custom Actor
3. ActorSystem Load Actor

3. Send/receive messages

• Encapsulating messages using sample classes
• SubmitTaskMessage - Submit Task Message
• SuccessSubmitTaskMessage - Task Submission Success Message
• Use an Actor similar to what you learned before, use! Send an asynchronous message

Reference Code

case class SubmitTaskMessage(msg:String)
case class SuccessSubmitTaskMessage(msg:String)

// Note: Actor under Akka will be imported
object SenderActor extends Actor {

  override def preStart(): Unit = println("implement SenderActor Of preStart()Method")

  override def receive: Receive = {
    case "start" =>
      val receiveActor = this.context.actorSelection("/user/receiverActor")
      receiveActor ! SubmitTaskMessage("Please complete#001 Task! ")
    case SuccessSubmitTaskMessage(msg) =>
      println(s"Received from ${sender.path}Message: $msg")
  }
}

object ReceiverActor extends Actor {

  override def preStart(): Unit = println("implement ReceiverActor()Method")

  override def receive: Receive = {
    case SubmitTaskMessage(msg) =>
      println(s"Received from ${sender.path}Message: $msg")
      sender ! SuccessSubmitTaskMessage("Complete Submission")
    case _ => println("Unmatched message type")
  }
}

object SimpleAkkaDemo {
  def main(args: Array[String]): Unit = {
    val actorSystem = ActorSystem("SimpleAkkaDemo", ConfigFactory.load())

    val senderActor: ActorRef = actorSystem.actorOf(Props(SenderActor), "senderActor")
    val receiverActor: ActorRef = actorSystem.actorOf(Props(ReceiverActor), "receiverActor")

    senderActor ! "start"
      
  }
}

Program output:

Received a message from akka://SimpleAkkaDemo/user/senderActor: Please complete the #001 task!
Received a message from akka://SimpleAkkaDemo/user/receiverActor: Complete submission

Akka Timer Task

What if we want to use the Akka framework to perform some tasks on a regular basis?

Usage

In Akka, a scheduler object is provided to implement the timer scheduling function.Using the ActorSystem.scheduler.schedule method, you can start a timed task.

The schedule method provides two forms of use for scala:

First: Send a message

def schedule(
    initialDelay: FiniteDuration,		// How late to start a timed task
    interval: FiniteDuration,			// How often to execute
    receiver: ActorRef,					// Which Actor to send a message to
    message: Any)						// Message to send
(implicit executor: ExecutionContext)	// Implicit parameters: manual import required

Second: Custom implementation

def schedule(
    initialDelay: FiniteDuration,			// How late to start a timed task
    interval: FiniteDuration				// How often to execute
)(f: ⇒ Unit)								// Functions that are executed periodically can be written here
(implicit executor: ExecutionContext)		// Implicit parameters: manual import required

Example 1

Example description

• Define an Actor that sends a message to the Actor every second, and the Actor prints the message when it receives it
• Using Send Message

Reference Code

 // 1. Create an Actor to receive and print messages
  object ReceiveActor extends Actor {
    override def receive: Receive = {
      case x => println(x)
    }
  }

  // 2. Build ActorSystem and load Actor
  def main(args: Array[String]): Unit = {
    val actorSystem = ActorSystem("actorSystem", ConfigFactory.load())
    val receiveActor = actorSystem.actorOf(Props(ReceiveActor))

    // 3. Start scheduler and send regular messages to Actor
    // Import an implicit conversion
    import scala.concurrent.duration._
    // Import Implicit Parameters
    import actorSystem.dispatcher

    actorSystem.scheduler.schedule(0 seconds,
      1 seconds,
      receiveActor, "hello")
  }

Example 2

Example description

• Define an Actor that sends a message to the Actor every second, and the Actor prints the message when it receives it
• Implement with Customization

Reference Code

object SechdulerActor extends Actor {
  override def receive: Receive = {
    case "timer" => println("Received Message...")
  }
}

object AkkaSchedulerDemo {
  def main(args: Array[String]): Unit = {
    val actorSystem = ActorSystem("SimpleAkkaDemo", ConfigFactory.load())

    val senderActor: ActorRef = actorSystem.actorOf(Props(SechdulerActor), "sechdulerActor")

    import actorSystem.dispatcher
    import scala.concurrent.duration._

    actorSystem.scheduler.schedule(0 seconds, 1 seconds) {
      senderActor ! "timer"
    }
  }
}

Implement communication between two processes

Case introduction

Send and receive messages between two processes based on Akka.Worker starts, connects to Master, sends a message, Master receives the message, and then replies to the Worker message.

[External chain picture transfer failed (img-RdkAKGbO-1569035711534)(assets/1552886264753.png)]

1. Worker implementation

step

1. Create a Maven module to import dependencies and configuration files
2. Create Startup WorkerActor
3. Send a "setup" message to WorkerActor, which receives a print message
4. Start test

Reference Code

Worker.scala

val workerActorSystem = ActorSystem("actorSystem", ConfigFactory.load())
val workerActor: ActorRef = workerActorSystem.actorOf(Props(WorkerActor), "WorkerActor")

// Send a message to WorkerActor
workerActor ! "setup"

WorkerActor.scala

object WorkerActor extends Actor{
  override def receive: Receive = {
    case "setup" =>
      println("WorkerActor:start-up Worker")
  }
}

2. Master implementation

step

1. Create a Maven module to import dependencies and configuration files
2. Create Start MasterActor
3. WorkerActor sends a "connect" message to MasterActor
4. MasterActor replies to the "success" message to WorkerActor
5. WorkerActor receives and prints received messages
6. Start Master, Worker tests

Reference Code

Master.scala

val masterActorSystem = ActorSystem("MasterActorSystem", ConfigFactory.load())
val masterActor: ActorRef = masterActorSystem.actorOf(Props(MasterActor), "MasterActor")

MasterActor.scala

object MasterActor extends Actor{
  override def receive: Receive = {
    case "connect" =>
      println("2. Worker connection to Master")
      sender ! "success"
  }
}

WorkerActor.scala

object WorkerActor extends Actor{
  override def receive: Receive = {
    case "setup" =>
      println("1. start-up Worker...")
      val masterActor = context.actorSelection("akka.tcp://MasterActorSystem@127.0.0.1:9999/user/MasterActor")

      // Send connect
      masterActor ! "connect"
    case "success" =>
      println("3. Connect Master Success...")
  }
}

Simple spark Communication Framework Case

Case introduction

Master and Worker communication simulation of Spark

• A Master
  • Manage Worker
• Several Workers (Workers can be added as needed)
  • register
  • Send a heartbeat

[External chain picture transfer failed (img-SRzZkVa3-1569035717187)(assets/1552890302701.png)]

Ideas for implementation

1. Building Master, Worker Stages
   • Build Master ActorSystem, Actor
   • Build Worker ActorSystem, Actor
2. Worker Registration Phase
   • The Worker process registers with Master (sends its ID, number of CPU cores, and memory size (M) to Master)
3. Worker sends heartbeat periodically
   • Worker periodically sends heartbeat messages to Master
4. Master Timed Heart Rate Detection Phase
   • Master periodically checks the heartbeat of Workers, removes some timed-out Workers, and sorts the Workers in reverse order by memory
5. Multiple Worker Test Stages
   • Start multiple Workers to see if registration is successful and stop a Worker to see if removal is correct

1. Engineering Setup

The project uses Maven to build the project

step

1. Build several projects separately

1. Import dependencies (pom.xml in the package)
   • master/worker adds common dependencies
2. Import the configuration file (application.conf in the package)
   • Modify Master's port to 7000
   • Modify Worker's port to 7100

2. Build Master and Worker

Build Master and Worker, respectively, and start testing

step

1. Create and load Master Actor
2. Create and load Worker Actor
3. Test if launch is successful

Reference Code

Master.scala

val sparkMasterActorSystem = ActorSystem("sparkMaster", ConfigFactory.load())
val masterActor = sparkMasterActorSystem.actorOf(Props(MasterActor), "masterActor")

MasterActor.scala

object MasterActor extends Actor{
  override def receive: Receive = {
    case x => println(x)
  }
}

Worker.scala

val sparkWorkerActorSystem = ActorSystem("sparkWorker", ConfigFactory.load())
sparkWorkerActorSystem.actorOf(Props(WorkerActor), "workerActor")

WorkerActor.scala

object WorkerActor extends Actor{
  override def receive: Receive = {
    case x => println(x)
  }
}

3. Worker Registration Phase Implementation

Send a registration message to Master when Worker starts

step

1. Worker sends registration messages to Master (workerid, number of cpu cores, memory size)
   • Randomly generate CPU cores (1, 2, 3, 4, 6, 8)
   • Random Generated Memory Size (512, 1024, 2048, 4096) (Unit M)
2. Master saves Worker information and replies to Worker's registration success message
3. Start test

Reference Code

MasterActor.scala

object MasterActor extends Actor{

  private val regWorkerMap = collection.mutable.Map[String, WorkerInfo]()

  override def receive: Receive = {
    case WorkerRegisterMessage(workerId, cpu, mem) => {
      println(s"1. Register NEW Worker - ${workerId}/${cpu}nucleus/${mem/1024.0}G")
      regWorkerMap += workerId -> WorkerInfo(workerId, cpu, mem, new Date().getTime)
      sender ! RegisterSuccessMessage
    }
  }
}

WorkerInfo.scala

/**
  * Work Node Information
  * @param workerId workerid
  * @param cpu CPU Number of Kernels
  * @param mem How much memory
  * @param lastHeartBeatTime Last Heart Rate Update Time
  */
case class WorkerInfo(workerId:String, cpu:Int, mem:Int, lastHeartBeatTime:Long)

MessagePackage.scala

/**
  * Registration Message
  * @param workerId
  * @param cpu CPU Number of Kernels
  * @param mem Memory size
  */
case class WorkerRegisterMessage(workerId:String, cpu:Int, mem:Int)

/**
  * Registration success message
  */
case object RegisterSuccessMessage

WorkerActor.scala

object WorkerActor extends Actor{

  private var masterActor:ActorSelection = _
  private val CPU_LIST = List(1, 2, 4, 6, 8)
  private val MEM_LIST = List(512, 1024, 2048, 4096)

  override def preStart(): Unit = {
    masterActor = context.system.actorSelection("akka.tcp://sparkMaster@127.0.0.1:7000/user/masterActor")

    val random = new Random()
    val workerId = UUID.randomUUID().toString.hashCode.toString
    val cpu = CPU_LIST(random.nextInt(CPU_LIST.length))
    val mem = MEM_LIST(random.nextInt(MEM_LIST.length))

    masterActor ! WorkerRegisterMessage(workerId, cpu, mem)
  }

  ...
}

4. Worker sends a heartbeat periodically

Worker sends a heartbeat message when Master returns to register successfully.When Master receives a heartbeat message from Worker, he needs to update the last heartbeat time of the corresponding Worker.

step

1. Write a tool class to read heartbeat send interval
2. Create a heartbeat message
3. Worker sends a heartbeat message regularly when it receives a successful registration
4. Master receives a heartbeat message to update Worker's last heartbeat time
5. Start test

Reference Code

ConfigUtil.scala

object ConfigUtil {
  private val config: Config = ConfigFactory.load()

  val `worker.heartbeat.interval` = config.getInt("worker.heartbeat.interval")
}

MessagePackage.scala

package com.itheima.spark.common

...

/**
  * Worker Heartbeat message
  * @param workerId
  * @param cpu CPU Number of Kernels
  * @param mem Memory size
  */
case class WorkerHeartBeatMessage(workerId:String, cpu:Int, mem:Int)

WorkerActor.scala

object WorkerActor extends Actor{
  ...

  override def receive: Receive = {
    case RegisterSuccessMessage => {
      println("2. Successfully registered to Master")

      import scala.concurrent.duration._
      import context.dispatcher

      context.system.scheduler.schedule(0 seconds,
        ConfigUtil.`worker.heartbeat.interval` seconds){
        // Send a heartbeat message
        masterActor ! WorkerHeartBeatMessage(workerId, cpu, mem)
      }
    }
  }
}

MasterActor.scala

object MasterActor extends Actor{
  ...

  override def receive: Receive = {
	...
    case WorkerHeartBeatMessage(workerId, cpu, mem) => {
      println("3. Heartbeat message received, Update last heartbeat time")
      regWorkerMap += workerId -> WorkerInfo(workerId, cpu, mem, new Date().getTime)
    }
  }
}

5. Master Timed Heart Rate Detection Phase

If a worker has not sent a heartbeat for more than a period of time, Master needs to remove the worker from the current worker collection.Heart beat timeout checking can be achieved through Akka's timer task.

step

1. Write a tool class to read check heartbeat interval interval, timeout
2. Check your heartbeat regularly and filter out Worker s that are longer than the timeout
3. Remove timeout Worker
4. Sort existing Workers in descending order by memory to print available Workers

Reference Code

ConfigUtil.scala

object ConfigUtil {
  private val config: Config = ConfigFactory.load()

  // Heart rate check interval
  val `master.heartbeat.check.interval` = config.getInt("master.heartbeat.check.interval")
  // Heart beat timeout
  val `master.heartbeat.check.timeout` = config.getInt("master.heartbeat.check.timeout")
}

MasterActor.scala

  override def preStart(): Unit = {
    import scala.concurrent.duration._
    import context.dispatcher

    context.system.scheduler.schedule(0 seconds,
      ConfigUtil.`master.heartbeat.check.interval` seconds) {
      // Filtered timeout worker
      val timeoutWorkerList = regWorkerMap.filter {
        kv =>
          if (new Date().getTime - kv._2.lastHeartBeatTime > ConfigUtil.`master.heartbeat.check.timeout` * 1000) {
            true
          }
          else {
            false
          }
      }

      if (!timeoutWorkerList.isEmpty) {
        regWorkerMap --= timeoutWorkerList.map(_._1)
        println("Remove Timeout worker:")
        timeoutWorkerList.map(_._2).foreach {
          println(_)
        }
      }

      if (!regWorkerMap.isEmpty) {
        val sortedWorkerList = regWorkerMap.map(_._2).toList.sortBy(_.mem).reverse
        println("Available Worker list:")
        sortedWorkerList.foreach {
          var rank = 1
          workerInfo =>
            println(s"<${rank}> ${workerInfo.workerId}/${workerInfo.mem}/${workerInfo.cpu}")
            rank = rank + 1
        }
      }
    }
  }
  ...
}

6. Multiple Worker Test Stages

Modify the configuration file and launch multiple worker s for testing.

step

1. Test whether starting a new Worker is registered successfully
2. Stop Worker to test if it can be removed from the existing list