Programmer Group

preface The Flink real-time computing platform has been built since half a year. Elasticsearch has been used in some storage layers and contacted Flink from scratch. In the past half a year, many pits have been encountered, which has changed from traditional development to big data development. Elasticsearch contains a variety of fuses to prev ...

Author: Tencent cloud flow computing Oceanus team ​ Introduction to flow computing Oceanus Stream computing Oceanus is a powerful tool for real-time analysis of big data product ecosystem. It is an enterprise level real-time big data analysis platform based on Apache Flink with the characteristics of one-stop development, seamless connection, s ...

background The flink cluster in the company has been running for a long time and has not been fully monitored. Recently, it plans to do this. Through online research, the deployment mode adopted by the company is per job mode. Finally, the job index is pushed to the push gateway of the intermediate gateway based on prometheus, Then prometheus ...

preface stay Flink of big data (Part I) In this paper, we introduce the characteristics, architecture, two-stage submission and data flow of Flink. This paper introduces the unique operator of Flink and the case of implementing WordCount with Flink 1, split and select operators The split operator splits a DataStream into two or more ...

This article is reproduced from the official account of "good future technology". The use of Flink CDC 2 is introduced in the case of Flink SQL, and the core design of CDC is interpreted. The main contents are as follows:caseCore designCode explanationIn August, Flink CDC released version 2.0.0. Compared with version 1.0, it supports ...

Flink execution plan level 1 - StreamTransformation The structure of list + linked list is constructed; Flink execution plan layer 2 - StreamGraph Transform the first layer into a graph structure; Next, it's time to convert StreamGraph to JobGraph. StreamGraph inherits the abstract class StreamingPlan and implements the getJobGraph method: /** ...

1, Localstreaminenvironment Localstreaminenvironment is a subclass of streaminexecutionenvironment. It runs programs locally, multithreaded, and in the JVM that instantiates localstreaminenvironment. It generates an embedded Flink cluster in the background and executes programs on the cluster. When instantiating this environment, it uses the de ...

Syllabus Course content learning effect Master the goal FlinkSQL FlinkTable master FlinkSQL master FlinkCEP FlinkCEP master Task performance optimization operator chain master slot sharing master Flink asynchronous IO master Checkpoint optimization master 1, Table & SQL (1) Overview Table API is a ...

The transformation operator we learned before cannot access the timestamp information and watermark information of the event. This is extremely important in some application scenarios. For example, map conversion operators such as MapFunction cannot access the timestamp or the event time of the current event. Based on this, the DataStream API ...

edition: Centos7.6 JDK1.8 Scala2.11 Python2.7 Git1.8.3.1 Apache Maven3.6.3 CDH6.3.2 Apache Flink1.12.0 The above software needs to be installed in advance!!! 1, Compile Flink 1 download the flink source code git clone https://github.com/apache/flink.git git checkout release-1.12.0 2. Add maven image Add the following mirrors in the m ...