Hadoop learning-hadoop installation

Stand-alone version configuration

Upload installation package

Upload to / bigdata

decompression

Unzip to / apps directory

tar -zxvf /bigdata/hadoop-2.7.1.tar.gz -C /apps

Configuring environment variables

vi /etc/profile
#In the final increase
export HADOOP_HOME=/apps/hadoop-2.7.1
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Loading new environment variables

source /etc/profile

test

hadoop jar /apps/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /input /output

Pseudo-Distributed Configuration

vi /apps/hadoop-2.7.1/etc/hadoop/hadoop-env.sh
#modify
export JAVA_HOME=/apps/jdk1.8.0_60

/apps/hadoop-2.7.1/etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop04:9000</value>
    </property>
</configuration>

/apps/hadoop-2.7.1/etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

Format file system

hdfs namenode -format

Start the file system

start-dfs.sh

See

jps

Fully distributed configuration

Installation preparation
Interoperability of Three Machine Networks
Firewall is closed and boot does not start
Configure host name and host mapping
All three machines need to install JDK (note version issues)
All three machines need ssh client installed
Mini1 can log in to mini1, mini2, mini3 password-free
Three machines need time synchronization

vi /apps/hadoop-2.7.1/etc/hadoop/hadoop-env.sh
#modify
export JAVA_HOME=/apps/jdk1.8.0_60

vi /apps/hadoop-2.7.1/etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop04:9000</value>
    </property>
    
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoopdata/tmp</value>
    </property>
    
    <property>
        <name>io.file.buffer.size</name>
        <value>4096</value>
    </property>
    
    <property>
        <name>fs.trash.interval</name>
        <value>5</value>
    </property>
     
    <property>
        <name>fs.trash.checkpoint.interval</name>
        <value>5</value>
    </property>
</configuration>

vi /apps/hadoop-2.7.1/etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    
    <property>
        <name>dfs.blocksize</name>
        <value>134217728</value>
    </property>
    
    <property>
        <name>dfs.http.address</name>
        <value>hadoop04:50070</value>
    </property>
    
    <!--To configure namenode Data Storage Path-->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/hadoopdata/dfs/name</value>
    </property>
    
    <!--To configure datanode Data Storage Path-->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/hadoopdata/dfs/data</value>
    </property>
    
    <!--Mirror file fsimage Detection catalogue-->
    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>/hadoopdata/dfs/cname</value>
    </property>
    <!--log file edits Detection catalogue-->
    <property>
        <name>dfs.namenode.checkpoint.edits.dir</name>
        <value>/hadoopdata/dfs/cname</value>
    </property>
    
    <property>
        <name>dfs.namenode.checkpoint.period</name>
        <value>3600</value>
    </property>
    
    <property>
        <name>dfs.namenode.checkpoint.txns</name>
        <value>1000000</value>
    </property>
    
    <!--Whether to open hdfs File system permissions-->
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
    
    <!--Whether to open webhdfs api Authority-->
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

vi /apps/hadoop-2.7.1/etc/hadoop/slaves

hadoop04
hadoop05
hadoop06

Distribution to two other machines

cd /apps
scp -r hadoop-2.7.1 hadoop05:$PWD
scp -r hadoop-2.7.1 hadoop06:$PWD

Formatting Cluster