php uses the ordered set zset of redis to implement delay queue

Delay queue is a message queue with delay function. Compared with ordinary queue, it can consume messages at a specified time. Application scenario of delay queue: 1. New users sign up, and send e-mails or on-site letters in 10 minutes. 2. After the user places an order, the order will be automatically voided if the payment is not made withi ...

Posted by DJP1986 on Sat, 21 Mar 2020 10:15:02 -0700

High availability of ES cluster

Preface: articles before us ELK+Kafka+Filebeat analysis Nginx log The configuration of ES cluster is mentioned in. In my later experiments, I found that although the cluster can be started and can process data normally, as long as the ES1 node is hung up, the whole cluster can not operate normally and t ...

Posted by ryanhowdy on Sun, 15 Mar 2020 23:18:03 -0700

ELK log collection

ELk introduction E: Elasticsearch stores data java 50: Logstash collection, filtering, forwarding, matching, large, slow start, middle role java K: Kibana filtering, analysis, graphic presentation java F: Filebeat collects logs and filters go Basic architecture diagram Traditional log analysis requirements (interview) 1. Find out the top ten ...

Posted by ppgpilot on Sun, 08 Mar 2020 03:42:58 -0700

Rsyslog return of the king

Rsyslog Everyone in the company is following the trend of learning es's log scheme. I've been struggling from three years ago to now. Filebeat + kafka + logstash+ es, at that time, I also wanted to build another storm. Later, I found that just a little log volume and a hammer kafka were used. Many of the tools that had been used were damaged by ...

Posted by lucie on Thu, 05 Mar 2020 23:02:10 -0800

Kafka principle and Kafka cluster deployment

Blog Outline:1, Overview of Kafka1) Message queuing2) Why use message queuing?3) What is Kafka?4) Characteristics of Kafka5) Kafka architecture6) Difference between Topic and Partition7) kafka flow chart8) Kafka's file storage mechanism9) Data reliability and persistence assurance10) leader election2, Deploy single Kafka1) Deploy Kafka2) Test ...

Posted by hanpedro on Thu, 27 Feb 2020 03:47:02 -0800

Flink notes: Flink data saving redis (custom Redis Sink)

This paper mainly introduces the process that Flink reads Kafka data and sinks (Sink) data to Redis in real time. Through the following link: Flink official documents , we know that the fault tolerance mechanism for saving data to Redis is at least once. So we use idempotent operation and the principle ...

Posted by quak on Wed, 19 Feb 2020 20:38:23 -0800

Flink notes: Flink uses EventTime to process real-time data

Combined with the previous post: Introduction to Time and Window in Flink , we learned that Flink's streaming process involves three time concepts. They are Event Time, induction time and Processing Time. If you still don't understand these three concepts of time, it's suggested that you jump to the pre ...

Posted by PHPSpirit on Wed, 12 Feb 2020 03:14:26 -0800

Spark Streaming of big data technology

Spark Streaming of big data technology 1: Overview 1. Definition: Spark Streaming is used for streaming data processing. Spark Streaming supports many data input sources, such as Kafka, Flume, Twitter, ZeroMQ and simple TCP sockets. After data input, you can use Spark's highly abstract primitives such ...

Posted by croakingtoad on Mon, 10 Feb 2020 07:28:21 -0800

Setting up Kafka-0.10.2 source reading environment and running Windows locally

Setting up Kafka-0.10.2 source reading environment and running Windows locally Version information 2. Building Kafka Source Environment 3. Configuring Kafka Source Environment Build a bin package Version information Kafka: 0.10.2,Scala: 2.10.6,Java: 1.8.0_221,IntelliJ IDEA: 2019.2,Zookeeper: 3.4 ...

Posted by crazydip on Thu, 30 Jan 2020 18:23:30 -0800

4.5 billion data migration records follow-up-to-day data volume levels of 10 million to clickhouse

4.5 billion data migration records follow-up-to-day data volume levels of 10 million to clickhouse Related Document Address flume Reference Address waterdrop Reference Address clickhouse Reference Address kafka Reference Address The environment log is on one server and the clickhouse cluster is on the other. thinking Previously, when m ...

Posted by gszauer on Mon, 20 Jan 2020 18:52:36 -0800