fault tolerance

End-to-end assurance is one of the key goals of structured flow design.

Structured Streaming sources,sinks, etc. are designed to track the exact processing progress and allow it to restart or rerun to handle any failures.

streaming source is a kafka-like offsets to track the read location of the stream. The execution engine uses checkpoints and write ahead logs to record the offset range values for each execution.

streaming sinks are designed to guarantee idempotency of processing

In this way, depending on the playback data source and streaming sinks, the structure flow achieves end-to-end guarantees with only one guarantee under any failure.

val lines = spark.readStream
  .format("socket")
  .option("host", "localhost")
  .option("port", 9999)
  .load()

// Split the lines into words
val words = lines.as[String].flatMap(_.split(" "))

// Generate running word count
val wordCounts = words.groupBy("value").count()

spark is SparkSession, lines is DataFrame, and DataFrame is Dataset[Row].

DataSet

Look at the code implementation of the trigger factor for Dataset, such as the foreach operation:

def foreach(f: T => Unit): Unit = withNewRDDExecutionId {

    rdd.foreach(f)

  }



 private def withNewRDDExecutionId[U](body: => U): U = {

    SQLExecution.withNewExecutionId(sparkSession, rddQueryExecution) {

      rddQueryExecution.executedPlan.foreach { plan =>

        plan.resetMetrics()

      }

      body

    }

  }

Then look at:

 def withNewExecutionId[T](

      sparkSession: SparkSession,

      queryExecution: QueryExecution,

      name: Option[String] = None)(body: => T): T = {

    val sc = sparkSession.sparkContext

    val oldExecutionId = sc.getLocalProperty(EXECUTION_ID_KEY)

    val executionId = SQLExecution.nextExecutionId

    sc.setLocalProperty(EXECUTION_ID_KEY, executionId.toString)

    executionIdToQueryExecution.put(executionId, queryExecution)

    try {     

      withSQLConfPropagated(sparkSession) {       

        try {         

          body

        } catch {         

        } finally {         

        }

      }

    } finally {

      executionIdToQueryExecution.remove(executionId)

      sc.setLocalProperty(EXECUTION_ID_KEY, oldExecutionId)

    }

  }

The real code to execute is queryExecution: QueryExecution.  

@transient private lazy val rddQueryExecution: QueryExecution = {

    val deserialized = CatalystSerde.deserialize[T](logicalPlan)

    sparkSession.sessionState.executePlan(deserialized)

  }

As you can see, it's session State. executePlan that executes logicalPlan and gets Query Execution.

The session State. executePlan here actually creates a QueryExecution object. Then executedPlan method of Query Execution is executed to get the physical plan of SparkPlan. How is it generated?

lazy val sparkPlan: SparkPlan = tracker.measurePhase(QueryPlanningTracker.PLANNING) {

    SparkSession.setActiveSession(sparkSession)   

    planner.plan(ReturnAnswer(optimizedPlan.clone())).next()

  }

Generated by planner.plan method.

planner is Spark planner. Defined in the BaseSessionStateBuilder class.

protected def planner: SparkPlanner = {

    new SparkPlanner(session.sparkContext, conf, experimentalMethods) {

      override def extraPlanningStrategies: Seq[Strategy] =

        super.extraPlanningStrategies ++ customPlanningStrategies

    }

  }

SparkPlanner class

SparkPlanner executes various strategies on LogicalPlan and returns the corresponding SparkPlan. For example, for streaming applications, there is a strategy: Data Source V2 Strategy.

Typical mapping relationships between logical plans and physical plans are as follows:

StreamingDataSourceV2Relation->ContinuousScanExec

StreamingDataSourceV2Relation->MicroBatchScanExec

The former corresponds to the case where there is no endOffset, and the latter corresponds to the case where there is an endOffset. The former is a continuous flow with no end, and the latter is a microbatch flow with intervals.

The delay of the former can reach 1 ms, and that of the latter can only reach 100 ms.

[Code]

case r: StreamingDataSourceV2Relation if r.startOffset.isDefined && r.endOffset.isDefined =>

      val microBatchStream = r.stream.asInstanceOf[MicroBatchStream]

      val scanExec = MicroBatchScanExec(

        r.output, r.scan, microBatchStream, r.startOffset.get, r.endOffset.get)

      val withProjection = if (scanExec.supportsColumnar) {

        scanExec

      } else {

        // Add a Project here to make sure we produce unsafe rows.

        ProjectExec(r.output, scanExec)

      }

      withProjection :: Nil

    case r: StreamingDataSourceV2Relation if r.startOffset.isDefined && r.endOffset.isEmpty =>

      val continuousStream = r.stream.asInstanceOf[ContinuousStream]

      val scanExec = ContinuousScanExec(r.output, r.scan, continuousStream, r.startOffset.get)

      val withProjection = if (scanExec.supportsColumnar) {

        scanExec

      } else {

        // Add a Project here to make sure we produce unsafe rows.

        ProjectExec(r.output, scanExec)

      }

      withProjection :: Nil