Curl command is a file transfer tool that uses URL rules to work on the command line. It supports the upload and download of files, so it is a comprehensive transmission tool, but according to tradition, curl is customarily called the download tool.

It supports many protocols including HTTP, HTTPS, ftp, etc. It also supports POST, cookies, authentication, downloading some files from specified offsets, user agent string, speed limit, file size, progress bar and other features.

Common Options

Crawl page information

-o Save the file as the file name specified on the command line
-O Use URL The default filename saves the file locally

 # Download the file locally and name it mygettext.html
curl -o mygettext.html http://www.gnu.org/software/gettext/manual/gettext.html

# Save the file locally and name it gettext.html. The following url can write rules
curl -O http://www.gnu.org/software/gettext/manual/gettext.html

# - O-O can download multiple files at the same time
# Without this option, it will print directly to standard output.

Web page redirection

Some webpages, such as www.sina.com, have jumped and can't get the source code of the webpage if curl is used directly. At this time, we need to add the - L option.

# When you encounter a redirection, use this option to redirect the request to a new address
curl -L www.sina.com

Continuous transmission of breakpoints

When curl page is in the middle of the terminal, you can use the - C option to follow the download that has been completed, and the downloaded files will not be re-downloaded.

# End the process before the file is downloaded
$ curl -O http://www.gnu.org/software/gettext/manual/gettext.html
##############             20.1%

$ curl -C -O http://www.gnu.org/software/gettext/manual/gettext.html
###############            21.1%

Access to request information or communication process

- i displays the header information of http response, along with the page code.
- I/- head displays only response header information.
- v shows the whole process of an http communication, including port connection and http request header information.
Or use the following command to get more detailed communication process:
curl --trace output.txt www.sina.com

Send Form Information

For the GET method, because the parameter data is on the url, curl can be used directly, which is also the default curl method.
For other methods, you need to specify with the - X option, such as POST, DELETE, etc.

$ curl -X POST --data "data=xxx" example.com

data is the same as - d and has the following usages:

-d @file # Put the submitted parameters in the file
-d "string" # The multi-parameter form is XXX & XXX
--data "string"
--data-ascii "string"
--data-binary "string"
--data-urlencode "string # url encoding for special symbols

Forgery of header information

- The e/- referer < URL > option can forge the source address.

# Pretend to jump from the http://www.google.com page to the destination page
$ curl --referer http://www.google.com http://man.linuxde.net

- The A/-user-agent <string> option can forge UA.

curl URL -A "Mozilla/5.0"

- H/--header custom header information

curl -H "Host:man.linuxde.net" -H "accept-language:zh-cn" <url>

- x/--proxy <host[:port]> Set up Agent

Setting cookie s

- The b/--cookie <name=val/file> option is used to set a cookie or to read cookie information from a specified file to initiate an http request.

$ curl --cookie "name=xxx;pass=xxx" www.example.com

- The c/--cookie-jar <file> option saves cookies to the specified file.

User authentication

- u/- user < user [: password]> authenticates http/ftp

Download files

$ curl -u name:password www.example.com
$ curl -O ftp://name:passwd@ip:port/demo/curtain/bbstudy_files/style.css  

Upload files

$ curl -T test.sql ftp://name:passwd@ip:port/demo/curtain/bbstudy_files/ 

Speed Limit and Quota

- limit-rate < rate > option to set transmission speed

curl URL --limit-rate 50k

- The max-filesize < bytes > option sets the maximum number of files downloaded

curl URL --max-filesize bytes

Reference resources

[1] Linux curl command in detail
[2]Detailed linux curl command and examples
[3]curl website development guide