Programmer Group

After building a high-availability HDFS cluster, Java can be used in Eclipse to operate HDFS and read and write files. High Availability HDFS Cluster Building Steps: https://blog.csdn.net/Chris_MZJ/article/details/83033471 Connecting HDFS with Eclipse 1. Place hadoop-eclipse-plugin-2.6.0.rar in the installation directory of ...

Reproduced from: https://www.cnblogs.com/MOBIN/p/5704001.html 1. Create HBase tables from Hive Create a Live table pointing to HBase using the HQL statement CREATE TABLE hbase_table_1(key int, value string) //Table name hbase_table_1 in Hive STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' //Designated Storage P ...

1 Spark Introduction Spark is a fast, universal and scalable large data analysis engine. It was born in AMPLab, University of California, Berkeley in 2009. It was open source in 2010. It became Apache incubator in June 2013 and top-level Apache project in February 2014. At present, Spark ecosystem has developed into a collecti ...

I. Installation and testing [root@VM_0_10_centos ~]# yum install subversion View svn version [root@VM_0_10_centos ~]# svnserve --version svnserve, version 1.7.14 (r1542130) compiled Apr 11 2018, 02:40:28 Copyright (C) 2013 The Apache Software Foundation. This software consists of contributions made by many people; see th ...

order This paper mainly studies Assignment Distribution Service of storm. AssignmentDistributionService storm-2.0.0/storm-server/src/main/java/org/apache/storm/nimbus/AssignmentDistributionService.java /** * A service for distributing master assignments to supervisors, this service makes the assignments notification * asynchronous. * * < ...

environmental information Fully Distributed Cluster (I) Cluster Foundation Environment and zookeeper-3.4.10 Installation and Deployment hadoop cluster installation configuration process You need to deploy hadoop cluster before installing hive Fully Distributed Cluster (II) Haoop 2.6.5 Installation and Deployment Hbase Cluster Installatio ...

Record problems with shiro at work. In fact, many problems are explained in official documents, but only limited to English reading ability. Only after solving the problem, can we find that it was explained. The shiro annotation @RequiresGuest, etc. is an intersection of permissions. Check order: RequiresRoles RequiresPermissions RequiresAu ...

Load and save function Data loading (json file, jdbc) and saving (json, jdbc) The test code is as follows: package cn.xpleaf.bigdata.spark.scala.sql.p1 import java.util.Properties import org.apache.log4j.{Level, Logger} import org.apache.spark.{SparkConf, SparkContext} import org.apache.spark.sql.{SQLContext, SaveMode} /* ...

1 Spark Operator 1.1 is divided into two categories 1.1.1 Transformation Transformation delays execution, which records metadata information and actually starts computing when the computing task triggers the Action. 1.1.2 Action 1.2 Two Ways to Create RDD RDD is created through the file system supported by HDFS. The ...

SparkSQL 1.x and 2.x programming API s have some changes that are used in enterprises, so both of them will use cases to learn. The case of using Spark SQL 1.x first IDEA+Maven+Scala 1. pom dependencies for importing SparkSQL In the previous Blog Spark case, the following dependencies are added to the pom dependencies of th ...