Programmer Group

Create kafka topic View all topic lists View the specified topic information Console to topic Production Data Console consumes topic data View the maximum (minimum) offset of a partition in topic Increase the number of topic partitions Delete topic, use cautiously, only delete metadata in zookeeper, message file must be deleted m ...

The front summarizes some pits of sqoop1, oozie and HBase under hue. When the project expires today, we must work out oozie workflow and schedule execution on time to skr skr. 1. The pits of sqoop mysql imported and exported from the front have been trampled. Later, it was found that besides cdh(5.15), sqoop1 was not automatically configured. ...

New table "apache_log" in hive CREATE TABLE apachelog ( host STRING, identity STRING, user STRING, time STRING, request STRING, status STRING, size STRING, referer STRING, agent STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = "([^ ] ...

Article Directory 1. Download the source code 2. Compile support for UDF 2.1 Upload and Unzip 2.2 Add UDF Function Class 2.3 Registration Functions 2.4 Compile hive 3. Deployment and installation 4. Testing UDF Friendly Tip: To reduce the probability of maven compilation errors on the server, you can choose to open the so ...

1. introduction: Install LZO: lzo is not native to Linux systems, so you need to download and install packages. There are at least three packages to install here. lzo, lzop, hadoop-gpl-packaging. Add index: The main function of gpl-packing is to create index for compressed lzo files. Otherwise, whether the compressed file is la ...

Hive storage format textfile hive's default storage format Storage: row storage Disk overhead, data parsing overhead Compressed text file hive cannot be merged and split SequenceFile Binary files are serialized into files in the form of key and value Storage: row storage Separable compression Generally selected block compression ...

In the work, the ID number in the Hive table needs to be strongly checked. Because the last bit is the checking bit, simple regularization can not be realized, and UDF is used to implement the relevant functions. I just realized the function, did not do in-depth optimization, welcome to leave a message, under the guidance of h ...

data base Create a database hive> CREATE DATABASE financials; hive> CREATE DATABASE IF NOT EXISTS financials; Display existing databases hive> SHOW DATABASES; default financials hive> CREATE DATABASE human_resources; hive> SHOW DATABASES; default financials human_resources Conditional Query Database hive> SHOW DATABASE ...

Installation and Use of Log Collection Framework Flume 1. Introduction to Flume 1.1. Overview of Flume Flume is a distributed, reliable and highly available system for collecting, transmitting and aggregating massive logs. Flume can collect files, socket data packets and other forms of source data. It can also output the collected d ...

After "$" corresponds to the Shell command of Linux, and after "hive >" corresponds to the command in CLI. CLI options - help to see the parameters of hive $hive --help --service cli usage: hive -d,--define <key=value> Variable subsitution to apply to hive commands. ...