Production tuning 1 HDFS core parameters

catalogue1 HFDS core parameters1.1 NameNode memory production configurationProblem descriptionConfiguration in hadoop-env.sh1.2 NameNode heartbeat concurrency configurationModify hdfs-site.xml configuration1.3 enable recycle bin configurationRecycle bin mechanismDescription of function parameters for opening recycle binStart Recycle Bin - modif ...

Posted by romic on Thu, 04 Nov 2021 15:44:50 -0700

Yarn capacity scheduler multi queue submission case

catalogueYarn capacity scheduler multi queue submission casedemandConfigure capacity scheduler for multiple queues1. Modify the following configurationUpload and download of SecureCRT2 upload to the cluster and distribute3 restart Yarn or yarn rmadmin -refreshQueues4 submit tasks to Hive queueSubmission method - jar packageTask priorityUse of t ...

Posted by neoform on Tue, 02 Nov 2021 06:55:51 -0700

[Hive] Chapter 5 DML data operation

Data import 1. Load data into the table (load) 1) Grammar hive> load data [local] inpath 'Data path' [overwrite] into table \ student [partition (partcol1=val1,...)]; (1) load data: indicates loading data (2) Local: indicates loading data from local to hive table; Otherwise, load data from HDFS to hive table (3) inpath: indicates the pa ...

Posted by PseudoEvolution on Sun, 31 Oct 2021 14:38:44 -0700

Hadoop is fully distributed

In fact, the construction methods of fully distributed and pseudo distributed are basically the same. Only individual files need to be changed. Let's start now! The first step is to configure the IP address, host name, and the mapping between host and IP address. The tutorial is included in my previous pseudo distributed building, which will n ...

Posted by smudge on Sun, 31 Oct 2021 12:30:57 -0700

Build records of problems encountered in native hadoop and cdh

hadoop startup error ERROR: Cannot set priority of datanode process The cluster was not formatted ./bin/hdfs namenode -format Modify yarn default port After that, it needs to be restarted because it is not hot database Error reporting 1045 [root@test ~]# systemctl stop mariadb ##Close mysql [root@test ~]# mysqld_safe --skip-grant-tables ...

Posted by dcooper on Thu, 28 Oct 2021 23:28:19 -0700

Hadoop: Flink on Yan service configuration and settings

Previously, I planned to directly install and configure the flink service on ambari to facilitate management, but I found that the flink integrated by ambari would have many problems, which would be inconvenient to manage (maybe I didn't find the correct method), so I planned to configure the service separately Download two files Flink-1.10.1 ...

Posted by sangamon on Tue, 26 Oct 2021 18:36:19 -0700

docker deploys Hadoop and HBase environment (Centos7 system)

1, Install docker 1. Download offline package Index of linux/static/stable/x86_64/ 2. Decompression tar -xzvf docker-18.06.3-ce.tgz (ce version means community free version, please specify   The difference between docker with ce and without ce) 3. Copy the extracted folder to the / usr/local directory cp docker-18.06.3-ce /usr/local ...

Posted by jenni on Tue, 26 Oct 2021 02:36:01 -0700

2021SC@SDUSC Hbase project overview

2021SC@SDUSC 1, HBase overview What is HBase HBase is a database system built on HDFS, which provides high reliability, high performance, column storage, scalability and real-time reading and writing. It is mainly used to store unstructured and semi-structured loose data. HBase uses hadoop HDFS as its file storage system, Hadoop MapReduc ...

Posted by gman-03 on Sun, 24 Oct 2021 20:13:50 -0700

More than running, everything updated | datart dormant in October, reborn

Hi Wen Ledao's large TV series "when is davinci updated" finally came to an end in the early autumn of 21 years, followed by the official release of the popular new product datart (Digital Art). This year, we can see that there are many such people in the group     So..   And so on..   What else?   Wha ...

Posted by xXx_Hobbes_xXx on Fri, 22 Oct 2021 00:16:58 -0700

It took a month to sort out this Hadoop blood spitting dictionary

  This document is compiled with reference to the official website of Hadoop and many other materials. For neat typesetting and comfortable reading, blurred and unclear pictures and black-and-white pictures are redrawn into high-definition color pictures. At present, Hadoop 2. X is widely used in enterprises, so this article focuses on Hado ...

Posted by b-real on Thu, 21 Oct 2021 21:42:54 -0700