Programmer Group

1, Notes Scala annotations use exactly the same as Java. Annotation is a good programming habit that a programmer must have. First sort out your thoughts through comments, and then reflect them in code. 1.1 single line notes package com.atguigu.bigdata.scala object ScalaComment{ def main(args: Array[String]): Unit = { // Si ...

1. Brief description of syntax /* object: Keyword to declare a singleton object (companion object) */ object HelloWorld{ /* main Method: the executed method can be called directly from the outside def Method name (parameter name: parameter type): return value type = {method body} */ def main(args: Array[String]):Unit = { println('H ...

object-oriented Scala's object-oriented thought is consistent with Java's object-oriented thought and concept.The syntax in Scala is different from that in Java, and more functions are added. Scala package Basic syntax: package nameThree functions of Scala package (same as Java) Distinguish between classes with the same nameWhen there ...

1, Spark cluster topology 1.1 cluster scale 192.168.128.10 master 1.5G ~2G Memory, 20 G Hard disk NAT,1~2 Nuclear; 192.168.128.11 node1 1G Memory, 20 G Hard disk NAT,1 nucleus 192.168.128.12 node2 1G Memory, 20 G Hard disk NAT,1 nucleus 192.168.128.13 node3 1G Memory, 20 G Hard disk NAT,1 nucleus 1.2 Spark installation mo ...

Proxy mode: Static proxyDynamic agent Before learning AOP, we need to understand the agent pattern Static proxy Static agent role analysis Abstract role: generally implemented using interfaces or abstract classes Real role: the role represented Agent role: agent real role; After representing a real role, you usually do some ancillary ...

Basic summary Spark is a fast, universal and scalable big data analysis engine. It is a big data parallel computing framework based on memory computing. Spark was born in the AMP laboratory at the University of California, Berkeley in 2009. It was open source in 2010 and became the top project of Apache in February 2014. This article is the f ...

1. Description Reader: Beginner SparkDevelopment environment: IDEA + spark2.4.8+jdk1.8.0_301Computer Configuration: 4 Core 8 ThreadView CPU methods: In Windows, type "wmic" in the cmd command, then enter "cpu get Name", "cpu get NumberOfCores", "cpu get NumberOfLogicalProcessors" in the ...

1, Run command bin/spark-submit \ --master yarn \ --deploy-mode cluster \ --class org.apache.spark.examples.SparkPi \ examples/jars/spark-examples_2.11-2.3.1.3.0.1.0-187.jar 2, Task submission flowchart 3, Startup script View the spark submit script file. The program entry is exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.depl ...

The Spark Computing Framework encapsulates three data structures for high concurrency and high throughput data processing in different application scenarios. RDD: Elastic Distributed DatasetAccumulator: Distributed shared write-only variablesBroadcast variables: distributed shared read-only variables RDD 1. What is RDD RDD (Resilient Di ...

window operation on data In the window operation of flow data, when event time passes the window time, it will close the window. Then in the actual production environment, due to various reasons such as the network, late data may appear, resulting in data disorder. As shown in the figure below, the ...