Programmer Group

SparkStreaming reads the Kafka data source and writes it to the Mysql database 1, Experimental environment The tools used in this experiment are kafka_2.11-0.11.0.2； zookeeper-3.4.5； spark-2.4.8； Idea； MySQL5.7 What is zookeeper? Zookeeper mainly serves distributed services, which can be used for unified configuration managemen ...

Actual combat of e-commerce offline warehouse project (Part 2) E-commerce analysis - core transactions 1, Business requirements Select indicators: order quantity, commodity quantity and payment amount, and analyze these indicators by sales region and commodity type. 2, Business database table structure 1. Relationship between databas ...

preface 1. How to analyze the method of high CPU utilization of JVM. 2. How to troubleshoot online jvm memory leaks. 3. How to locate the jvm thread deadlock problem. 4. How to solve the problem of jvm frequently full gc, etc. These problems will be solved step by step through this article. 1, Pre preparation 1. First prepare a web s ...

Explore Kafka from the perspective of PHP - realize simple producer function code first public function actionProducer() { $conf = new \RdKafka\Conf(); $conf->set('metadata.broker.list', 'broker address'); /** * kafka Delivery callback * RdKafka\Message Object * ( * [err] => 0 // If it is equal to 0, the delivery is ...

Introduction to Kafka Kafka is a popular distributed message subscription system. In addition to Kafka, there are MQ, Redis, etc. We can regard the message queue as a pipeline. The two ends of the pipeline are the message producer and the message consumer respectively. After the message producer generates logs and other messages, it can be s ...

1, Introduction to KerberosKerberos can put the authenticated key on a reliable node before cluster deployment. When the cluster is running, the nodes in the cluster are authenticated with the key, and the nodes that pass the authentication can provide services. The node attempting to impersonate cannot communicate with the nodes in the cluster ...

The most used go client of kafka should be sarama, but the old version of sarama did not support the consumption mode of consumer groups, so most people use sarama cluster. Later, sarama supported the consumption mode of consumer groups, and sarama cluster stopped maintenance. However, there are few online analysis of sarama consumer groups, a ...

origin The company wants to make a product. Of course, the environment is built before making the product. Most of the environments have been written before. Today, let's write about the installation of kafka. Step 1: download the installation package Download the kafka installation package. Mine is kafka_2.11-2.1.1. However, there are s ...

Spark 2.0 has produced a new stream processing framework Structured Streaming, which is a scalable and fault-tolerant stream processing engine built on Spark SQL Engine. Using Structured Streaming, you can perform streaming computing on static data (Dataset/DataFrame) like batch computing. With the continuous arrival of data, Spark SQL Engine w ...

catalogue 1, About zookeeper . 1 overview of zookeeper 2. Definition of Zookeeper   3. Working mechanism of Zookeeper   4. Features of Zookeeper 5. Data structure of zookeeper 2, Application scenario of Zookeeper 3, Zookeeper's election mechanism 1. The electoral mechanism was launched for the first time 2. It is not the f ...