tcp socket communication optimization and troubleshooting under Linux

@Date 2018.12.09

I. optimization

1. TPC receiving window

• Problem: when the receiving window queue of TCP is blocked - > the sender continues to send - > the receiver loses - > the sender retransmits - > the network becomes bad
• Solution: the receiver tells the sender the size of the receive cache - > the receive cache is full - > the sender cannot send

# Increase receive window cache size
net.ipv4.tcp_rmem = "40960 873800 41943040"
net.core.rmem_max = 41943040
net.core.rmem_default = 873800

# Open win scale
net.ipv4.tcp_window_scaling = 1

2. TCP congestion window

#Optimize the initial size of congestion window

3. Recovery of time "wait status

# Adjust the recovery time of time "wait
$ vi /etc/sysctl.conf

net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30

net.core.somaxconn = 2048
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 10000
net.core.netdev_max_backlog = 20000

net.ipv4.tcp_rmem = 7168 11264 16777216
net.ipv4.tcp_wmem = 7168 11264 16777216
net.ipv4.tcp_mem = 786432 2097152 3145728
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_orphans = 131072
net.ipv4.tcp_max_tw_buckets=180000
fs.file-max = 1000000

Two. Problem

1. Connection timeout

• Question:

# Check whether there are a large number of requests with different codes, too many connections, Syn queue overflows, and connections are discarded
$ netstat -anp | grep SYN_RECV/TIME_WAIT

• Solution: change the number of configuration files and open syncookie

• Question:

# The number in front has been growing rapidly. A large number of requests from clients cause the accept queue to be full, and then syn packets will be discarded
# When the server receives syn, it first looks at the syn queue, then at the accept queue. If there is a full syn, it discards it
$ netstat -s | grep -i listen

• Solution: increase the length of accept queue -- configuration file, net.core.somaxconn=8192. Calculation formula: Len of accept queue = min (backlog + 1, somaxconn)

• Problem: clients often fail to connect

#Quad: source ip, destination ip, source port, destination port
 #A client can only use fixed port range when connecting to a server
 #socket in time ﹣ wait state cannot be reused

• Solution: client solution, modify the socket configuration file

# Transfer port usage range
$ --net.ipv4.ip_local_port_range="1024 65535"
# Multiplexing time out status port
$ --net.ipv4.tcp_tw_reuse=1
$ net.ipv4.tcp_timestamp =1
# Speed up time out status port release
$ net.ipv4.tcp_tw_recyle=1
$ net.ipv4.tcp_timestamp=1

2. too many open files

# The user program does not call the close function and will not release automatically -- program exception
$ netsata -anp | grep CLOSE_WAIT