Programmer Group

Delay queue is a message queue with delay function. Compared with ordinary queue, it can consume messages at a specified time. Application scenario of delay queue: 1. New users sign up, and send e-mails or on-site letters in 10 minutes. 2. After the user places an order, the order will be automatically voided if the payment is not made withi ...

Preface: articles before us ELK+Kafka+Filebeat analysis Nginx log The configuration of ES cluster is mentioned in. In my later experiments, I found that although the cluster can be started and can process data normally, as long as the ES1 node is hung up, the whole cluster can not operate normally and t ...

ELk introduction E: Elasticsearch stores data java 50: Logstash collection, filtering, forwarding, matching, large, slow start, middle role java K: Kibana filtering, analysis, graphic presentation java F: Filebeat collects logs and filters go Basic architecture diagram Traditional log analysis requirements (interview) 1. Find out the top ten ...

Rsyslog Everyone in the company is following the trend of learning es's log scheme. I've been struggling from three years ago to now. Filebeat + kafka + logstash+ es, at that time, I also wanted to build another storm. Later, I found that just a little log volume and a hammer kafka were used. Many of the tools that had been used were damaged by ...

Blog Outline:1, Overview of Kafka1) Message queuing2) Why use message queuing?3) What is Kafka?4) Characteristics of Kafka5) Kafka architecture6) Difference between Topic and Partition7) kafka flow chart8) Kafka's file storage mechanism9) Data reliability and persistence assurance10) leader election2, Deploy single Kafka1) Deploy Kafka2) Test ...

This paper mainly introduces the process that Flink reads Kafka data and sinks (Sink) data to Redis in real time. Through the following link: Flink official documents , we know that the fault tolerance mechanism for saving data to Redis is at least once. So we use idempotent operation and the principle ...

Combined with the previous post: Introduction to Time and Window in Flink , we learned that Flink's streaming process involves three time concepts. They are Event Time, induction time and Processing Time. If you still don't understand these three concepts of time, it's suggested that you jump to the pre ...

Spark Streaming of big data technology 1: Overview 1. Definition: Spark Streaming is used for streaming data processing. Spark Streaming supports many data input sources, such as Kafka, Flume, Twitter, ZeroMQ and simple TCP sockets. After data input, you can use Spark's highly abstract primitives such ...

Setting up Kafka-0.10.2 source reading environment and running Windows locally Version information 2. Building Kafka Source Environment 3. Configuring Kafka Source Environment Build a bin package Version information Kafka: 0.10.2,Scala: 2.10.6,Java: 1.8.0_221,IntelliJ IDEA: 2019.2,Zookeeper: 3.4 ...

4.5 billion data migration records follow-up-to-day data volume levels of 10 million to clickhouse Related Document Address flume Reference Address waterdrop Reference Address clickhouse Reference Address kafka Reference Address The environment log is on one server and the clickhouse cluster is on the other. thinking Previously, when m ...