Programmer Group

background In traditional bank IT architecture, online trading and statistical analysis systems often use different technologies and physical devices to migrate online trading data to the analysis system through ETL which is executed regularly. As a data service resource pool, the same data may be accessed by different types of microservices. ...

Catalog   Preface 1. What is a windowing function 2. Window-opening function syntax 3. Classification of window-opening functions 4. Cumulative Statistics Window Opening Function 4.1 Cumulative sum(xx) over 4.2 Cumulative Average avg(xx) ...

Installation of Hive libraries for large data components: 1. First, download it from the official website of hive. https://hive.apache.org/downloads.html Note that when choosing the hive version, you need to pay attention to the hadoop version. The hive must be based on hadoop before it can be started. The following error is reported in the ...

Super simple centos7 configuration Hadoop 2.7.7 + flume 1.8.0 Introduction of flume: https://blog.csdn.net/qq_40343117/article/details/100119574 1 - Download the installation package Download address: http://www.apache.org/dist/flume/ Choose the ...

3. File storage format 1. Column Storage and Row Storage (1) Features of column storage: Because the data aggregated storage for each field can greatly reduce the amount of data read when only a few fields are needed for query. Each field must ...

When we use Spark, we mainly use it to process large quantities of data quickly. So what data sources will we have in actual development and production? I summarize them as follows: text csv json parquet jdbc hive kafka elasticsearch Next, all the tests are based on the spark local mode, be ...

- Restore content to start --- 1: tuple: Similar to lists, you can store multiple values, multiple data types, but the difference is that the ancestor itself cannot be modified 1 Purpose: Record multiple values, tuples are more appropriate when multiple values have no changed requirements 2 Definition: Compared to the list type, just [] is ...

HQL Data Operation The source of our content is Hadoop Mass Data Processing Technology Details and Project Practice, People's Posts and Telecommunications Publishing House 1. Loading data LOAD DATA INPATH '/user/hadoop/o' INTO TABLE test; If the test table is a partition table, specify a distinction in HQL: LOAD DATA INPATH '/USER/HA ...

Installation Prerequisites Hadoop cluster has been installed and configured (single or fully distributed) Software Download Hive: https://hive.apache.org/index.html Hive Installation Configuring environment variables Upload the downloaded Live package to the machine and extract it to the specified path Edit / etc/profile to configure Hi ...

Hive analysis function and window function Supported after Hive 0.11, scan multiple input rows to calculate the results of each row. Usually used with OVER,PARTITION BY, ORDER BY, WINDOWING. Unlike traditional grouping results, traditional results have only one result in each group. The results of the analysis function will appear many times, ...