preface
Today, let's play ChaosMesh to simulate the network duplicate package. At the same time, we should also look at the direct impact on the application.
target
Simulate network duplicate packets.
to configure
yaml file configuration
[root@s5 ChaosMesh]# cat network-duplicate.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: network-duplicate
namespace: chaos
spec:
action: duplicate
mode: one
selector:
labelSelectors:
"k8s.kuboard.cn/name": "svc-7dmall"
duplicate:
duplicate: "40"
correlation: "25"
duration: "10s"
scheduler:
cron: '@every 15s'
Interface configuration:
implement
On the command line:
[root@s5 ChaosMesh]# kubectl apply -f network-duplicate.yaml networkchaos.chaos-mesh.org/network-duplicate created
After the configuration is completed and submitted on the interface, the execution starts.
verification
- Enter the simulated POD, check the rules added on qdisc, and then execute tcpdump packet capture.
- see qdisc Rules on [root@svc-7dmall-664d59f75b-whtvc /]# tc qdisc ls dev eth0 qdisc netem 1: root refcnt 2 limit 1000 duplicate 40% [root@svc-7dmall-664d59f75b-whtvc /]# - implement tcpdump Grab bag [root@svc-7dmall-664d59f75b-whtvc /]# tcpdump -i eth0 -w networkduplicate.cap tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes ^C195 packets captured 195 packets received by filter 0 packets dropped by kernel [root@svc-7dmall-664d59f75b-whtvc /]#
From the above results, 195 bags were caught.
- Enter the machine where the pod is located or use other tools to copy the captured files locally.
[root@s7 ~]# docker cp 3d4ab3241d8a:/networkduplicate.cap ./
- Open view with wireshark.
There are indeed a large number of duplicate packages.
The normal packet capture results are as follows:
- Start a thread to access the application with Jmeter for 60s. See the difference.
Results with duplicate packages:
Results without duplicate packages:
From the above results, the impact on performance is not small when duplicate packets are generated.
- The direct feeling of the application is that the response time is long and the TPS decreases.
- The user's direct feeling is: slow but responsive or slow until an error is reported.
- Enter the simulated POD and view the rules on qdisc again
[root@svc-7dmall-664d59f75b-whtvc /]# tc qdisc ls dev eth0 qdisc noqueue 0: root refcnt 2 [root@svc-7dmall-664d59f75b-whtvc /]#
recovery
Stop the case on the interface
Stop the case with the command line
[root@s5 ChaosMesh]# kubectl delete -f network-duplicate.yaml networkchaos.chaos-mesh.org "network-duplicate" deleted [root@s5 ChaosMesh]#
Retransmission principle logic description and RTO calculation process
There are many reasons for duplicate packets, such as application failure, network equipment failure, service downtime and so on.
Let's mainly explain the logic of retransmission.
The retransmission timer determines the message retransmission mechanism. Its function is to maintain the retransmission timeout.
After sending the message, the retransmission timer starts and stops after receiving the ACK. If no ack is received, the sender considers that the message is lost and retransmitted, and the RTO is doubled; If the ACK is not received after 2 times RTO, it will be retransmitted again.
In Linux, the maximum number of retransmissions is 15 by default, which is controlled by two parameters:
net.ipv4.tcp_retries1 = 3 net.ipv4.tcp_retries2 = 15
- tcp_retries1 = 3 means that after three retransmissions, if no ACK is received, the route will be updated first in subsequent retransmissions.
- tcp_retries2 = 15 refers to retransmission up to 15 times. If the retransmission fails for 15 times, you will give up.
Normally, you can end it when you see here. However, it does not rule out that some people want to understand the calculation logic of RTO. Then keep looking.
Maybe you'll ask how long the RTO is? In the Linux source code, there is such a definition (the source path includes / net / TCP. H, which is 134 and 135 lines in my 3.10 kernel code):
#define TCP_RTO_MAX ((unsigned)(120*HZ)) #define TCP_RTO_MIN ((unsigned)(HZ/5))
See here, we know that RTO has maximum and minimum values, which are HZ/5 and 120*HZ respectively. But there are questions. What is the value of HZ?
You can execute the following commands in the system.
[root@s5 ~]# cat /boot/config-`uname -r` | grep '^CONFIG_HZ=' CONFIG_HZ=1000
In my system, the median value is 1000, that is, the minimum value of RTO is 200 and the maximum value is 120000 (in milliseconds).
But how is this RTO calculated? This value is determined by__ tcp_set_rto (lines 606-609 in the 3.10 source code path include/net/tcp.h) is calculated by the function (I won't expand the calculation logic of srtt. The more I say, the easier it is to mess up. Those interested can check the data themselves).
static inline u32 __tcp_set_rto(const struct tcp_sock *tp)
{
return (tp->srtt >> 3) + tp->rttvar;
}
The value calculated by this function cannot be less than TCP_ RTO_ The value of min.
Who determines the maximum RTO? That's tcp_bound_rto · (lines 600-604 in the 3.10 source code path include/net/tcp.h).
static inline void tcp_bound_rto(const struct sock *sk)
{
if (inet_csk(sk)->icsk_rto > TCP_RTO_MAX)
inet_csk(sk)->icsk_rto = TCP_RTO_MAX;
}
The above is the calculation logic of RTO.
Leave room for thinking
- How to analyze the cause of network packet retransmission?
- Is there a function to set the minimum RTO?