Programmer Group

environmental information Fully Distributed Cluster (I) Cluster Foundation Environment and zookeeper-3.4.10 Installation and Deployment hadoop cluster installation configuration process You need to deploy hadoop cluster before installing hive Fully Distributed Cluster (II) Haoop 2.6.5 Installation and Deployment Hbase Cluster Installatio ...

I. overview Concept: It is a reliable, scalable and distributed open source software. It is a framework that allows large data and distributed processing across clusters of computers, using a simple programming model (mapreduce) It can be extended from a single server to thousands of hosts, and each node provides computing ...

Article directory 1. Basic information 2. Installation process 1. Switch to hadoop account and decompress hadoop to the destination installation directory by tar-zxvf command: 2. Create tmpdir directory: Configure hadoop-env.sh file: 4. Configure mapred-env.sh file: 5. Configure the core-site.xml file core-site.xml Configure ...

1 Spark Operator 1.1 is divided into two categories 1.1.1 Transformation Transformation delays execution, which records metadata information and actually starts computing when the computing task triggers the Action. 1.1.2 Action 1.2 Two Ways to Create RDD RDD is created through the file system supported by HDFS. The ...

Following hadoop fully distributed installation After that, zookper+hbase security is combined. Continue with the previous environment configuration. I. zookper Installation 1.1 Download and Unzip Software cd /software wget -c http://apache.fayea.com/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz tar -zxf zookeeper-3.4.1 ...

Today, 42 jobs were enabled when using hive to perform mr. Halfway through the execution, the following errors were reported suddenly. I have never encountered them before. I don't know if it is the reason why there are too many jobs. The error prompt was that port 10020 could not be accessed from the host. Check the reason on ...

As a framework of log real-time collection, flume can be connected with SparkStreaming real-time processing framework. Flume generates data in real-time and sparkStreaming does real-time processing. Spark Streaming docks with FlumeNG in two ways: one is that FlumeNG pushes the message Push to Spark Streaming, the other is that S ...

YARN The construction of yarn Cluster planning To configure Test case wordcount Use the test case wordcount provided by MapReduce The construction of yarn Cluster planning To configure Modify the configuration file mapred-sitex.xml <property> <name>mapreduce.framework.name</name> <value& ...

To access Hive, Java needs to connect Hive by beeline. hiveserver2 provides a new command-line tool, beeline. hiveserver2 upgrades the previous hive, which has more powerful functions. It adds permission control. To use beeline, you need to start hiverserver2 first, and then use beeline connection. Operation steps: (1) Modi ...

1. Edit profile file export JAVA_HOME=/usr/lib/jvm/jdk/ export HADOOP_INSTALL=/sur/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin export PATH=$PATH:$HADOOP_INSTALL/sbin export HADOOP_MAPRED_HOME=$HADOOP_INSTALL export HADOOP_COMMON_HOMR=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL export YARN_HOME-$HADOOP_INSTAL ...