Programmer Group

Requirement: read out the data in hive and write it into es. Environment: spark 2.0.2 1. enableHiveSupport() is set in sparksession SparkConf conf = new SparkConf().setAppName("appName").setMaster("local[*]"); SparkSession spark = SparkSession .builder() .appName("Java Spark SQL basic exam ...

In AWS, Athena can be used to analyze the logs saved in S3. He can convert the logs into the format of database tables, so that they can be queried through sql statements. This function is similar to using logparser to analyze Exchange or IIS logs on a windows Server. Let's do a demonstration, record the management log through Cloudtrail, and ...

Overview of Hive Hive is a Hadoop-based data warehouse tool that maps structured data files to a table and provides SQL-like query capabilities. Hive is essentially a MapReduce program that converts HQL. Data processed by Hive is stored in HDFS, and the underlying implementation of analytic data can be MapReduce, tes, or Spark, with its executo ...

Download Hive installation package from website Modify hive-site.xml (version 2.3.4 does not have this file to create directly) <?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>javax.jdo.option. ...

Business logic processing Peers To judge whether it is a peer object, we can judge whether two objects have passed through multiple identical places by using longitude and latitude. Of course, each monitoring device can also be marked. When an object passes through the monitoring device, it can be captured by the device. Tra ...

I. add hive-site.xml Add the hive-site.xml configuration file under $SPARK_HOME/conf in order to access hive metadata normally vim hive-site.xml <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.1.201:3306/hiveDB?createDatabaseIfNotExist=true ...

Hot issues of hbase data: The solution is to preprocess the rowkey of the hot data, add some prefixes, and distribute the hot data to multiple region s. Pre merger? Dynamic partition? At the beginning of the initial data, the data should be partitioned, stored in different region s, and load balanced. Example: for example, it is easy to divide ...

Import data (cluster as object) In Sqoop, the concept of "import" refers to the transfer of data from non big data cluster (RDBMS) to big data cluster (HDFS, HIVE, HBASE). It is called "import", that is, using the import keyword. 1 RDBMS to HDFS 1) make sure the Mysql service is turned on normally 2) create a new table i ...

I. installation Hive installation: brew install hive mysql installation: brew install mysql Start MySQL: bash mysql.server start 2. Metabase configuration Hive uses derby as the metabase by default. Here we use mysql to store metadata, and do some initialization configuration below Log in to mysql: mysql -u root ...

I. Introduction The window function row ou number() is a function that groups one field and then takes the first values sorted by another field, which is equivalent to grouping topN. If a windowing function is used in the SQL statement, the SQL statement must be executed with HiveContext. II. Code practice [use HiveContext] package big.dat ...