Parallel programming of Scala Actor in spark notes

Keywords: Big Data Scala React Programming Java

1.1. Course objectives
1.1.1. Objective 1: familiar with Scala Actor concurrent programming
1.1.2. Goal 2: prepare for Akka learning

Note: Scala Actor is scala 2.10.x and earlier.

Scala added Akka as its default Actor in version 2.11.x, and the old one has been abandoned.

1.2. What is Scala actor 1.2.1. Concept

The Actor in scala can realize the powerful function of parallel programming. It is a concurrency mechanism based on the event model. Scala uses the sending and receiving of messages to achieve high concurrency.

Actors can be regarded as independent entities, and there is no relationship between them. However, they can communicate through messages. When an Actor receives information from other actors, it can make various responses as needed. The type of message can be arbitrary, and the content of the message can be arbitrary.

1.2.2. Differences between java Concurrent Programming and Scala Actor programming

As for Java, we all know that its multithreaded implementation needs to use synchronized keyword for code block synchronization, object lock mutual exclusion, etc. for shared resources (variables, objects, etc.). And often a big try... It is a headache to add wait method, notify method and notifyAll method to catch statement block. The reason is that most of Java uses variable state object resources. When sharing these resources to realize multithreaded programming, it is very important to control the resource competition and prevent the object state from being accidentally modified, and the invariance of the object state is also difficult to guarantee.

Unlike Java's thread model based on shared data and lock, Scala's actor package provides another model that does not share any data and relies on message passing, so as to carry out concurrent programming.

1.2.3. Execution sequence of Actor

1. First call the start() method to start the Actor

2. After calling the start() method, its act() method will be executed.

3. Send message to Actor

4. After the act method is executed, the program will call the exit method.

1.2.4. Method of sending message

Note: Future represents the result state of an asynchronous operation, and may not be the result of an asynchronous task actually completed

Any is the superclass of all classes, and the generic of future any is the type of asynchronous operation result.

1.3. Actor actual combat 1.3.1. The first example

How to realize concurrent programming of actor:

1. Define a class or object to inherit the character of Actor. Pay attention to the import scala.actors.Actor.

2. Rewrite the corresponding act method

3. Call the start method of Actor to execute Actor.

4. When the act method is completed, the whole program is finished.

package cn.itcast.actor

 

import scala.actors.Actor

 

  

 

object Actor1 extends Actor{

  //Override act method

  def act(){

    for(i <- 1 to 10){

      println("actor-1 " + i)

      Thread.sleep(2000)

    }

  }

}

 

object Actor2 extends Actor{

  //Override act method

  def act(){

    for(i <- 1 to 10){

      println("actor-2 " + i)

      Thread.sleep(2000)

    }

  }

}

 

object ActorTest extends App{

  //Start Actor

  Actor1.start()

  Actor2.start()

}
—

Note: the above calls the start() method of two singleton objects respectively. Their act() method will be executed. The same as opening two threads in java, the run() method of the thread will be executed.

Note: the two actors are executed in parallel. After the for loop in the act() method is executed, the actor program exits.

1.3.2. Second example

How to realize sending and receiving messages by actor

1. Define a class or object to inherit the character of Actor. Pay attention to the import scala.actors.Actor.

2. Rewrite the corresponding act method

3. Call the start method of Actor to execute Actor.

4. Send messages to actor s in different ways

5. In act method, receive method is used to receive and process messages.

6. After the act method is executed, the program exits.

package cn.itcast.actor

import scala.actors.Actor

class MyActor extends Actor {

 

  override def act(): Unit = {

      receive {

        case "start" => {

          println("starting ...")

        }

      }

    }

  }

}

 

 

object MyActor {

  def main(args: Array[String]) {

    val actor = new MyActor

    actor.start()

    actor ! "start"

  

    println("Message sending completed!")

 

  }

}

1.3.3. The third example

How to implement actor s to continuously receive messages:

In the act method, you can use the while(true) method to continuously accept messages.

package cn.itcast.actor

import scala.actors.Actor

class MyActor1 extends Actor {

 

  override def act(): Unit = {

    while (true) {

      receive {

        case "start" => {

          println("starting ...")

        }

        case "stop" => {

          println("stopping ...")

        }

      }

    }

  }

}

 

 

object MyActor1 {

  def main(args: Array[String]) {

    val actor = new MyActor1

    actor.start()

    actor ! "start"

    actor ! "stop"

 

  }

}

Note: when (true) loop is added to act() method to receive messages continuously

Note: sending the start message and stop message is asynchronous, but the process of the Actor receiving the message is synchronous and executed in order

1.3.4. Fourth example

Use react method instead of receive method to accept messages

Advantage: react mode can reuse threads and avoid frequent thread creation, destruction and switching. More efficient than receive

Note: if you want to execute message processing repeatedly, the outer layer of react should use loop instead of while.

package cn.itcast.actor

import scala.actors.Actor

class YourActor extends Actor {

  override def act(): Unit = {

    loop {

      react {

        case "start" => {

          println("starting ...")

        }

        case "stop" => {

          println("stopping ...")

 

        }

      }

    }

  }

}

 

  

 

 

object YourActor {

  def main(args: Array[String]) {

    val actor = new YourActor

    actor.start()

    actor ! "start"

    actor ! "stop"

    println("Message sending completed!")

 

  }

}

1.3.5. Fifth example

Combining case class sample class to send and receive messages

1. Encapsulate the message in a sample class

2. Perform different operations by matching different sample classes

3. The Actor can return messages to the sender. Returns a message to the sender of the current message through the sender method

package cn.itcast.actor

import scala.actors.Actor

 

 

case class SyncMessage(id:Int,msg:String)//Synchronous message

case class AsyncMessage(id:Int,msg:String)//Asynchronous message

case class ReplyMessage(id:Int,msg:String)//Return result message

 

class MsgActor extends Actor{

  override def act(): Unit ={

    loop{

      react{

        case "start"=>{println("starting....")}

 

        case SyncMessage(id,msg)=>{

          println(s"id:$id, SyncMessage: $msg")

          Thread.sleep(2000)

          sender !ReplyMessage(1,"finished...")

        }

        case AsyncMessage(id,msg)=>{

          println(s"id:$id,AsyncMessage: $msg")

         // Thread.sleep(2000)

          sender !ReplyMessage(3,"finished...")

          Thread.sleep(2000)

        }

 

      }

    }

  }

}

 

object MainActor {

  def main(args: Array[String]): Unit = {

     val mActor=new MsgActor

        mActor.start()

        mActor!"start"

 

        //Synchronization message has return value

     val reply1= mActor!?SyncMessage(1,"I'm syncing")

      println(reply1)

      println("===============================")

        //Asynchronous no return message

     val reply2=mActor!AsyncMessage(2,"I am asynchronous no return message")

  

      println("===============================")

        //Asynchronous with return message

    val reply3=mActor!!AsyncMessage(3,"I have asynchronous return message")

    //Future's apply() method will build an asynchronous operation and return a value at some point in the future

      val result=reply3.apply()

      println(result)

 

  }

}

1.3.6. Practice actual combat

Demand:

Write a stand-alone WordCount with actor concurrent programming, take multiple files as input, summarize multiple tasks after calculation, and get the final result.

General thought steps:

1. Continuously receive messages through loop +react

2. Use case class sample class to match corresponding operations

3. scala provides the interface Source for file reading, and calls its fromFile method to get the file content.

4. Summarize the number of words in each file and store them in a ListBuffer.

5. Finally, the results in ListBuffer are summarized globally.

package cn.itcast.actor

import java.io.File

import scala.actors.{Actor, Future}

import scala.collection.mutable

import scala.io.Source

 

  

 

case class SubmitTask(fileName: String)

case class ResultTask(result: Map[String, Int])

class Task extends Actor {

  override def act(): Unit = {

    loop {

      react {

        case SubmitTask(fileName) => {

          val contents = Source.fromFile(new File(fileName)).mkString

          val arr = contents.split("\r\n")

          val result = arr.flatMap(_.split(" ")).map((_, 1)).groupBy(_._1).mapValues(_.length)

          //val result = arr.flatMap(_.split(" ")).map((_, 1)).groupBy(_._1).mapValues(_.foldLeft(0)(_ + _._2))

          sender ! ResultTask(result)

        }

      }

    }

  }

}

 

object WorkCount {

  def main(args: Array[String]) {

    val files = Array("d://aaa.txt", "d://bbb.txt","d://ccc.txt")

    val replaySet = new mutable.HashSet[Future[Any]]

    val resultList = new mutable.ListBuffer[ResultTask]

    for(f <- files) {

      val t = new Task

      val replay = t.start() !! SubmitTask(f)

      replaySet += replay

    }

    while(replaySet.size > 0){

      val toCumpute = replaySet.filter(_.isSet)

      for(r <- toCumpute){

        val result = r.apply()

        resultList += result.asInstanceOf[ResultTask]

        replaySet.remove(r)

      }

 

    }

    val finalResult = resultList.map(_.result).flatten.groupBy(_._1).mapValues(x => x.foldLeft(0)(_ + _._2))

    println(finalResult)

  }

}

Posted by bogu on Wed, 16 Oct 2019 01:13:40 -0700