Programmer Group

catalogue preface Part I: Transformation operator Value type map() map mapPartitions() executes Map in partition units mapPartitionsWithIndex() with partition number flatMap() flatten glom() partition conversion array groupBy() group Extension: complex wordcount Filter sample() distinct() de duplication coalesce() repartition rep ...

1. Brief description of syntax /* object: Keyword to declare a singleton object (companion object) */ object HelloWorld{ /* main Method: the executed method can be called directly from the outside def Method name (parameter name: parameter type): return value type = {method body} */ def main(args: Array[String]):Unit = { println('H ...

Hive environment construction Hive engine introduction Hive engine includes: default MR, tez, spark Hive on Spark: hive not only stores metadata, but also is responsible for SQL parsing and optimization. The syntax is HQL syntax. The execution engine has become Spark, and Spark is responsible for RDD execution. Spark on hive: hive is only ...

Data skew caused by Shuffle When data skew occurs during Shuffle, we generally follow the troubleshooting steps ① Check the WEB-UI page to check the execution of tasks in the Stage of each Job, and whether there is an obvious situation that the execution time is too long ② If the task reports an error, check the corresponding log excepti ...

Spark 2.0 has produced a new stream processing framework Structured Streaming, which is a scalable and fault-tolerant stream processing engine built on Spark SQL Engine. Using Structured Streaming, you can perform streaming computing on static data (Dataset/DataFrame) like batch computing. With the continuous arrival of data, Spark SQL Engine w ...

1, Spark cluster topology 1.1 cluster scale 192.168.128.10 master 1.5G ~2G Memory, 20 G Hard disk NAT,1~2 Nuclear; 192.168.128.11 node1 1G Memory, 20 G Hard disk NAT,1 nucleus 192.168.128.12 node2 1G Memory, 20 G Hard disk NAT,1 nucleus 192.168.128.13 node3 1G Memory, 20 G Hard disk NAT,1 nucleus 1.2 Spark installation mo ...

Basic summary Spark is a fast, universal and scalable big data analysis engine. It is a big data parallel computing framework based on memory computing. Spark was born in the AMP laboratory at the University of California, Berkeley in 2009. It was open source in 2010 and became the top project of Apache in February 2014. This article is the f ...

1. Description Reader: Beginner SparkDevelopment environment: IDEA + spark2.4.8+jdk1.8.0_301Computer Configuration: 4 Core 8 ThreadView CPU methods: In Windows, type "wmic" in the cmd command, then enter "cpu get Name", "cpu get NumberOfCores", "cpu get NumberOfLogicalProcessors" in the ...

1, Run command bin/spark-submit \ --master yarn \ --deploy-mode cluster \ --class org.apache.spark.examples.SparkPi \ examples/jars/spark-examples_2.11-2.3.1.3.0.1.0-187.jar 2, Task submission flowchart 3, Startup script View the spark submit script file. The program entry is exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.depl ...

The Spark Computing Framework encapsulates three data structures for high concurrency and high throughput data processing in different application scenarios. RDD: Elastic Distributed DatasetAccumulator: Distributed shared write-only variablesBroadcast variables: distributed shared read-only variables RDD 1. What is RDD RDD (Resilient Di ...