[cluster building topic] - Kafka cluster building

Kafka -- cluster building Service discovery: first of all, the mutual discovery of services between kfakas is realized by using Zookeeper. Therefore, Zookeeper must be built in advance. To build a Zookeeper cluster, please pay attention to the blog's article: Building Zookeeper cluster;Service relat ...

Posted by Bauer418 on Mon, 29 Jun 2020 20:50:55 -0700

Big data Hadoop cluster construction

Big data Hadoop cluster construction 1, Environment Server configuration: CPU model: Intel ® Xeon ® CPU E5-2620 v4 @ 2.10GHz CPU cores: 16 Memory: 64GB operating system Version: CentOS Linux release 7.5.1804 (Core) Host list: IP host name 192.168.1.101 node1 192.168.1.102 node2 1 ...

Posted by SL-Cowsrule on Sun, 21 Jun 2020 17:57:54 -0700

Flink flow processing API code details, including a variety of Source, Transform, Sink cases, Flink learn

Hello, everyone, I am later, I will share my learning and work in the drips, hope that I have an opportunity to help you some articles, all articles will be launched in the official account, welcome to my official account "later X big data", thank you for your support and recognition. It's another week without change. I went back to Y ...

Posted by GamingWarrior on Tue, 16 Jun 2020 20:35:17 -0700

spark structured streams, creating streaming DataFrame s and streaming Datasets

Create Streaming DataFrame and Streaming Datasets Streaming DataFrames are available through SparkSession.readStream() The returned DataStreamReader interface (Scala / Java / Python document) is created. Input Sources Common built-in Sources File source: Reads files from a specified directory as st ...

Posted by IwnfuM on Sun, 07 Jun 2020 18:13:27 -0700

Kafka Quick Start - Kafka Cluster Deployment

Kafka Quick Start (6) - Kafka Cluster Deployment I. Kafka Cluster Deployment Plan 1. Operating system selection In general, production environments should deploy Kafka clusters on Linux operating systems for the following reasons:(1) At the bottom of the Kafka client, Java selector is used. The implementation mechanism of selector on Linux is ...

Posted by colesw on Mon, 25 May 2020 11:24:58 -0700

Kafka Core API - Stream API

Kafka Stream concepts and initial high-level architecture diagrams Kafka Stream is a new Feature introduced by Apache Kafka from version 0.10 that provides streaming and analysis of data stored in Kafka.In short, Kafka Stream is a class library for stream computing, similar to Storm, Spark Streaming, Flink, but much lighter. Basic concepts of ...

Posted by Dujo on Sun, 24 May 2020 09:39:16 -0700

[DNS] Ubuntu 14.04 deploy local DNS Service

1, Background In the Intranet environment, you need to install the local DNS service at many times. Take the ubuntu 14.04 64 bit operating system environment as an example to illustrate the DNS server building method. 2, Deployment steps 2.1 install bind software # apt-get install bind9 -y 2.2 edit profile A. add a custom domain name reso ...

Posted by ramas on Tue, 05 May 2020 18:03:14 -0700

Site online linkage ELK log entry

ELK schema log processing logic: 1. When Filebeat is installed in the business layer, it will automatically obtain the domain name and environment information of the host running site, add channel and env tags, and take the value of channel as the topic information of kafka 2. Kafka receives the new field and Topic information of Filebeat, and ...

Posted by BluntedbyNature on Tue, 05 May 2020 17:52:59 -0700

Go Language (18) context & Log Items

Context&log items context Method to cancel goroutine in general scenarios var wg sync.WaitGroup var exit bool func worker(exitChan chan struct{}) { LOOP: for { fmt.Printf("work\n") time.Sleep(time.Second) /*if exit { break } */ select { case <- exitChan: break LOOP ...

Posted by loopykd on Mon, 04 May 2020 16:40:51 -0700

java implementation of spark streaming and kafka integration for flow computing

java implementation of spark streaming and kafka integration for flow computing Added on June 26, 2017: took over the search system, and in the past half year, I have gained a lot of new experience. I'm lazy to change this vulgar article. Let's take a look at this new blog to understand the following vulgar code, http://blog.csdn.net/yujishi2 ...

Posted by rednax on Thu, 30 Apr 2020 23:48:23 -0700