The Flink real-time computing platform has been built since half a year. Elasticsearch has been used in some storage layers and contacted Flink from scratch. In the past half a year, many pits have been encountered, which has changed from traditional development to big data development. Elasticsearch contains a variety of fuses to prev ...
Posted by banzaimonkey on Mon, 06 Dec 2021 15:50:31 -0800
Author: Tencent cloud flow computing Oceanus team
Introduction to flow computing Oceanus
Stream computing Oceanus is a powerful tool for real-time analysis of big data product ecosystem. It is an enterprise level real-time big data analysis platform based on Apache Flink with the characteristics of one-stop development, seamless connection, s ...
Posted by jassh on Sat, 04 Dec 2021 13:49:59 -0800
The flink cluster in the company has been running for a long time and has not been fully monitored. Recently, it plans to do this. Through online research, the deployment mode adopted by the company is per job mode. Finally, the job index is pushed to the push gateway of the intermediate gateway based on prometheus, Then prometheus ...
Posted by jtgraphic on Fri, 26 Nov 2021 16:43:59 -0800
stay Flink of big data (Part I) In this paper, we introduce the characteristics, architecture, two-stage submission and data flow of Flink. This paper introduces the unique operator of Flink and the case of implementing WordCount with Flink
1, split and select operators
The split operator splits a DataStream into two or more ...
Posted by hoogeebear on Mon, 22 Nov 2021 07:22:57 -0800
This article is reproduced from the official account of "good future technology". The use of Flink CDC 2 is introduced in the case of Flink SQL, and the core design of CDC is interpreted. The main contents are as follows:caseCore designCode explanationIn August, Flink CDC released version 2.0.0. Compared with version 1.0, it supports ...
Posted by wxflint on Wed, 10 Nov 2021 23:25:41 -0800
Flink execution plan level 1 - StreamTransformation The structure of list + linked list is constructed;
Flink execution plan layer 2 - StreamGraph Transform the first layer into a graph structure;
Next, it's time to convert StreamGraph to JobGraph. StreamGraph inherits the abstract class StreamingPlan and implements the getJobGraph method:
Posted by tetecko81sk on Wed, 10 Nov 2021 09:49:48 -0800
Localstreaminenvironment is a subclass of streaminexecutionenvironment. It runs programs locally, multithreaded, and in the JVM that instantiates localstreaminenvironment.
It generates an embedded Flink cluster in the background and executes programs on the cluster.
When instantiating this environment, it uses the de ...
Posted by capbiker on Tue, 09 Nov 2021 00:10:46 -0800
The transformation operator we learned before cannot access the timestamp information and watermark information of the event. This is extremely important in some application scenarios. For example, map conversion operators such as MapFunction cannot access the timestamp or the event time of the current event.
Based on this, the DataStream API ...
Posted by byronbailey on Mon, 01 Nov 2021 22:39:01 -0700
edition: Centos7.6 JDK1.8 Scala2.11 Python2.7 Git18.104.22.168 Apache Maven3.6.3 CDH6.3.2 Apache Flink1.12.0
The above software needs to be installed in advance!!!
1, Compile Flink
1 download the flink source code
git clone https://github.com/apache/flink.git
git checkout release-1.12.0
2. Add maven image
Add the following mirrors in the m ...
Posted by mustatin on Mon, 01 Nov 2021 04:52:28 -0700