Hadoop: Flink on Yan service configuration and settings

Previously, I planned to directly install and configure the flink service on ambari to facilitate management, but I found that the flink integrated by ambari would have many problems, which would be inconvenient to manage (maybe I didn't find the correct method), so I planned to configure the service separately Download two files Flink-1.10.1 ...

Posted by sangamon on Tue, 26 Oct 2021 18:36:19 -0700

Submit the sql script file with Flink sql client

Title: submit sql script file with flynk sql client Date: twenty 21-10-22 22:11:34 Tag: [Flink, SQL client] Category: flink We know that sql-client.sh can provide us with an SQL interactive interface, so that we can see the execution results or interactive query table results without executing an SQL. In fact, we can also submit sql scripts t ...

Posted by Drannon on Fri, 22 Oct 2021 07:39:30 -0700

Flink source code note 02 - generating StreamGraph from DataStream

StreanGraph It is a DAG (stored in the form of adjacency table), which stores the topology information of the whole flow and is composed of a series of StreamEdge and StreamNode. Call StreamExecutionEnvironment.getStreamGraph() to generate a StreamGraph. StreamExecutionEnvironment.getStreamGraph() clears the transformations saved in StreamExe ...

Posted by dr_overload on Sun, 10 Oct 2021 20:32:39 -0700

Restrictions on Flink DataStream Java Lambda expressions

​ Original text: Restrictions on Flink DataStream Java Lambda expressions Java 8 introduces some new language features aimed at faster and clearer coding. With the most important feature 'lambda expression', it opens the door to functional programming. Lambda expressions allow functions to be implemented and passed directly without declaring ...

Posted by system_critical on Fri, 08 Oct 2021 11:30:18 -0700

[vernacular analysis] Flink's Watermark mechanism

catalogue 0x00 summary 0x01 problem Q1. What are the common requirements / solutions in Flink stream processing applications Question 2. Watermark should be translated into water mark Question 3. What is the essence of watermark Question 4. How does watermark solve the problem 0x02 background concept Stream processing Out of order 0 ...

Posted by schilly on Tue, 28 Sep 2021 20:06:11 -0700

FlinkSQL custom function (UDF) dimension table conversion

preface Relationship between Table and SQL: SQL is the inheritance encapsulation of Table (this point is in Flink concept For example, StreamTableEnvironment can be embodied by inheriting from TableEnvironment. Therefore, the use of tables in official documents can be reflected in SQL, such as user-defined functions, Table API & custom fun ...

Posted by scofansnags on Tue, 07 Sep 2021 19:38:53 -0700

Kakfa Flink hive integration principle and actual combat code

Hello, I'm brother Tu. At present, he works as a big data algorithm engineer in a large Internet factory. Today, a fan sent a message in the group asking if there is any information about Flink DDL, saying that he is a new learner. In order to let this fan quickly learn how to use Flink DDL, I will explain the principle and attach the actu ...

Posted by amazinggrace1983 on Tue, 07 Sep 2021 16:02:28 -0700