Configuration and startup of Hadoop pseudo distributed environment

1. Environmental preparation On a linux machine, install the hadoop running environment. For the installation method, see: Establishment of HADOOP operation environment 2. Start HDFS and run MapReduce 2.1. Configure cluster 1. Configuration: hadoop-env.sh Get the installation path of JDK in Linux system: [root@ hadoop101 ~]# echo $JAVA_HOME /op ...

Posted by spooke2k on Tue, 25 Feb 2020 19:31:27 -0800

Talk about discovery group of artemis

order This paper focuses on the discovery group of artemis DiscoveryGroup activemq-artemis-2.11.0/artemis-core-client/src/main/java/org/apache/activemq/artemis/core/cluster/DiscoveryGroup.java public final class DiscoveryGroup implements ActiveMQComponent { private static final Logger logger = Logger.getLogger(DiscoveryGroup.class); pri ...

Posted by andreas on Tue, 11 Feb 2020 08:51:47 -0800

Troubleshooting of namenode in HDFS

After NameNode fails, you can use the following two methods to recover data: 1. Copy data in SecondaryNameNode to the directory where NameNode stores data (1) kill -9 NameNode process [test@hadoop151 ~]$ jps 3764 DataNode 4069 NodeManager 3654 NameNode 7738 Jps [test@hadoop151 ~]$ kill -9 3654 [test@h ...

Posted by webtechatlantic on Mon, 03 Feb 2020 08:51:44 -0800

Talk about sendDelay of Carrera Producer

order This article mainly studies sendDelay of Carrera Producer ProducerInterface DDMQ/carrera-sdk/producer/java/carrera-producer-sdk/src/main/java/com/xiaojukeji/carrera/producer/ProducerInterface.java public interface ProducerInterface { void start() throws Exception; void shutdown(); Result sendMessage(Message message); R ...

Posted by theorok on Mon, 06 Jan 2020 02:39:07 -0800

Kafka ou 2.12-2.0.0 installation process

1. Download Kafka? 2.12-2.0.0.tgz and upload it to / usr/local/kafka / Extract and remove the installation package 2. configuration Modify the config/server.properties of each server cd /usr/local/kafka/kafka_2.12-2.0.0/config vim server.properties Modify several places:broker.id : unique, fill in the number, 10,11,12,13,1 ...

Posted by N350CA on Sun, 15 Dec 2019 12:29:42 -0800

Various daemon States

Various daemon States Article directory Various daemon States 1. pseudo distribution 2. Fully distributed 2.1 Do not open yarn 2.2 Open yarn (MRAppMaster and YarnChild are available only with MapReduce tasks) 3. Fully Distributed (High Availability) 3.1 Manual Disaster Recovery 3.1.1 No yarn 3 ...

Posted by palpie on Sat, 05 Oct 2019 17:26:13 -0700

Hadoop 3.2.0 Source Analysis: Default Container Executor for Container and Linux Container Executor

Container is optional in yarn. This paper introduces the following two types: DefaultContainerExecutor  LinuxContainerExecutor  Controlled by configuration parameters: yarn.nodemanager.container-executor.class   When NodeManager initializes, load org.apache.hadoop.yarn.server.nodemanager.NodeManager#serviceInit     // ...

Posted by Atanu on Mon, 22 Apr 2019 22:00:35 -0700

YDB is deployed in Mini Centos environment based on haoop, zookeeper and kafka

YDB is deployed in Mini Centos environment based on haoop, zookeeper and kafka YDB introduction YDB full name Yan Yun YDB It is a real-time, multi-dimensional, interactive query, statistics and analysis engine based on Hadoop distributed architecture. It has second-level performance under trillion data scale and stable and reliable perform ...

Posted by narked on Wed, 13 Feb 2019 13:12:18 -0800

Yarn tuning

1.Yarn Common Commands: [rachel@bigdata-senior01 bin]$ ./yarn Usage: yarn [--config confdir] COMMAND where COMMAND is one of: resourcemanager run the ResourceManager nodemanager run a nodemanager on each slave timelineserver run the timeline server rmadmin admin tools version ...

Posted by soloslinger on Sun, 03 Feb 2019 08:15:18 -0800

Hadoop fully distributed deployment

I. overview Concept: It is a reliable, scalable and distributed open source software. It is a framework that allows large data and distributed processing across clusters of computers, using a simple programming model (mapreduce) It can be extended from a single server to thousands of hosts, and each node provides computing ...

Posted by spyke01 on Fri, 01 Feb 2019 03:57:15 -0800