Today, 42 jobs were enabled when using hive to perform mr. Halfway through the execution, the following errors were reported suddenly. I have never encountered them before. I don't know if it is the reason why there are too many jobs. The error prompt was that port 10020 could not be accessed from the host.

Check the reason online: This is because DataNode needs to access MapReduce JobHistory Server, its default port is 0.0.0:10020.

Error reporting information:

java.io.IOException: java.net.ConnectException: Call From hadoop01/192.168.80.128 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:343)
	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428)
	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:568)
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:323)
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:320)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
	at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:338)
	at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:603)
	at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:631)
	at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:289)
	at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)
	at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437)
	at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1650)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1409)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Solution:

1) Open jobhistory in the / etc/hadoop/mapred-site.xml configuration file

<property>

        <name>mapreduce.jobhistory.address</name>

<! - Configure the actual host name and port - > and

        <value>hadoop01:10020</value>

 </property>

2) Increase the heap size of history server, depending on the specific situation.  

Under the configuration file ${HADOOP_DIR}/etc/hadoop/mapred-env.sh:

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=2000

3) Open history server

[root@hadoop01 sbin]# mr-jobhistory-daemon.sh start historyserver