wget bind ip failure

Keywords: DNS Mobile lsof Linux

Causes:

Our program needs to pull files from the file server (across the computer room).
The machines running the program are telecommunications, interconnection and mobile lines, and three IP lines are assumed to be
01_DX_IP, 01_LT_IP, 01_YD_IP
File servers are also telecommunications, unicom, Mobile 3 lines, 3 IP
02_DX_IP, 02_LT_IP, 02_YD_IP
Its domain name is myfile.com

Use wget to pull out files and bind the export IP to ensure that the export bandwidth is used as much as possible.

wget  --bind-address=01_YD_IP  -t 3 -T 120  --limit-rate=3M -S -O 1407856770354_858932.mp4 http://myfile.com/data10/sony/303/2014-08/12/1407856770354_858932.mp4

According to the pre-thought, bound to mobile IP, the normal DNS server will return a file server's mobile IP, the speed should not be too slow, in case of things, download speed is less than 100k.

1. Investigation:

1.1 Observation File Download Connection

Looking at the log printed by wget, it shows that the IP of Unicom is actually accessed when the connection is established.

Resolving myfile.com... 02_LT_IP,  ...
Connecting to myfile.com|02_LT_IP|:80... connected.
HTTP request sent, awaiting response... ^C

Use lsof to observe process connections

[xxx@c3n~]$ sudo lsof -p  133396
COMMAND    PID USER   FD   TYPE     DEVICE    SIZE/OFF        NODE NAME
wget    133396 root  cwd    DIR        8,2        4096      
59640871425 /datapool/log/inject/xxx.log
wget    133396 root    6w   REG       0,17   164815000 59756490241 /datapool/injectBase/1408203875738_777462.mp4
wget    133396 root    7u  IPv4 1479434406         0t0         TCP 01_YD_IP:29657->02_LT_IP:http (ESTABLISHED)

Note that this line TCP 01_YD_IP: 29657 - > 02_LT_IP: http (ESTABLISHED)
Local bind address are already in effect and have been bound to mobile IP, but the IP address of the file server is Unicom IP, which can only be a problem with DNS parsing.

1.2 Tracking DNS Resolution

wget adds the - d parameter to view the details of wget

Setting --bind-address (bindaddress) to 01_YD_IP
Setting --tries (tries) to 3
Setting --timeout (timeout) to 120
Setting --limit-rate (limitrate) to 3M
Setting --server-response (serverresponse) to 1
Setting --output-document (outputdocument) to 1407856770354_858932.mp4
DEBUG output created by Wget 1.12 on linux-gnu.

--2017-07-11 11:08:37--  http://myfile.com/data10/sony/303/2014-08/12/1407856770354_858932.mp4
Resolving myfile.com... 02_LT_IP, ...
Caching myfile.com => 02_LT_IP
Connecting to myfile.com|02_LT_IP|:80... Releasing 0x0000000002618870 (new refcount 0).
Deleting unused 0x0000000002618870.
connected.
Created socket 4.
Releasing 0x0000000002618580 (new refcount 1).

---request begin---
GET /data10/sony/303/2014-08/12/1407856770354_858932.mp4 HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: xxxxx
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... ^C

You can see that wget first performs DNS parsing, then actually establishes a connection with the file server and downloads the files.

For port 53, the communication between observer and DNS server

[xxx@c3n]# tcpdump port 53 -i p3p1
11:14:34.187939 IP 01_LT_IP.22683 > pdns.dnspod.cn.domain: 47926+ A? myfile.com. (50)
11:14:34.217628 IP pdns.dnspod.cn.domain > 01_LT_IP.22683: 38811 5/0/0 A 02_LT_IP (130)

Here you can see that the exit IP requesting DNS service is a Unicom IP01_LT_IP. No wonder the DNS server parses myfile.com-> 02_LT_IP.

1.3 Summary

At this point, it's clear that wget downloads files in two steps
1. Request DNS server to get IP address of file server
2. Connect with file server and download files
In Step 1, there is no bind IP, which results in a wrong line for DNS parsing. The direct intersection of different operators has a narrow export, so the download speed can not go up.

2. Solution

This is obviously a problem with the wget tool. Try to look at the data and see if the new version of wget has solved this problem.
Two new parameters have been added since v1.18, which are used to bind the exit IP when requesting a DNS server

'–bind-dns-address=ADDRESS'
[libcares only] This address overrides the route for DNS requests. If you ever need to circumvent the standard settings from /etc/resolv.conf, this option together with '–dns-servers' is your friend. ADDRESS must

./configure --with-cares

Posted by dirgeshp on Fri, 14 Jun 2019 18:19:54 -0700