Original collation is not easy, please indicate the source of reprinting: java Learning collation for apache commons HttpClient sending get and post requests

Code download address: http://www.zuidaima.com/share/1754065983409152.htm

HttpClient is something I want to study recently. Some of the applications I've thought about before haven't been implemented very well. After discovering this open source project, it's a bit of an eyebrow. There's still a way to solve the headache cookie problem. I sorted out some things on the Internet and wrote them well. I put them here.

HTTP protocol is probably the most important protocol used on the Internet nowadays, more and more. Java Applications need to access network resources directly through HTTP protocol. Although in JDK's java .NET The basic functions of accessing the HTTP protocol have been provided in the package, but for most applications, the functions provided by the JDK library itself are not rich and flexible enough. HttpClient Apache Jakarta Common is a sub-project to provide efficient, up-to-date, feature-rich client-side programming toolkit supporting HTTP protocol, and it supports the latest version and recommendations of HTTP protocol. HttpClient has been used in many projects, such as Apache Jakarta and two other well-known open source projects, Cactus and HTMLUnit, which use HttpClient. See more applications using HttpClient. http://wiki.apache.org/jakarta-httpclient/HttpClientPowered. HttpClient The project is very active and there are a lot of people using it. The current version of HttpClient is 3.0 RC4 released on October 11, 2005.

－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

Applying HttpClient to deal with various stubborn WEB servers
Change from: http://blog.csdn.net/ambitiontan/archive/2006/01/06/572171.aspx

In general, we use IE or Navigator browser to access a WEB server, to browse the page to view information or submit some data and so on. Some of these pages are just ordinary pages, some need user login before they can be used, or need authentication and some are transmitted by encryption, such as HTTPS. The browsers we use today will not be a problem in dealing with these situations. But sometimes you may need to use programs to access some of these pages, such as "stealing" some data from other people's pages; using pages provided by some sites to accomplish certain functions, such as saying that we want to know where a mobile phone number belongs and we don't have such data ourselves, so we have to use existing websites of other companies to accomplish this task. Yes, at this time we need to submit our mobile phone number to the web page and parse the data we want from the returned page. If the other party is only a very simple page, then our program will be very simple, this article is not necessary to waste words here. However, considering some service authorization issues, many companies often provide pages that can not be accessed through a simple URL, but must be registered and logged in before they can use the pages that provide services. At this time, COOKIE issues are involved. We know that current popular dynamic web technologies such as ASP and JSP process session information through COOKIE. In order for our program to use the service pages provided by others, we need the program to log in first and then visit the service pages. This process requires cookies to be processed by itself. Think about when you use java. .Net What a horrible thing to do when HttpURLConnection completes these functions! And that's just a common "stubbornness" in what we call stubborn WEB servers! What about uploading files over HTTP? No headache, these problems can be easily solved with "it"!

 

We can't list all the possible stubbornness. We'll deal with some of the most common problems. Of course, as mentioned earlier, if we use java.net.HttpURLConnection to solve these problems by ourselves, it's terrible. So before we start, let's introduce an open source project, httpclient in Apache Open Source Organization, which belongs to Jakarta's Commons project. The current version is 2.0RC2. There is already a sub-project of net under commons, but HTTP client is put forward separately, so it is not easy to access http server.

Commons-httpclient project is specially designed to simplify the communication programming between HTTP client and server. It can make it easy to solve the original headaches now, for example, if you no longer care about HTTP or HTTPS communication mode, tell it you want to use HTTPS mode, the rest of the things to be done by httpclient for you. This article will introduce how to use httpclient to solve several problems we often encounter when we write HTTP client programs. In order to familiarize readers with this project more quickly, we first give a simple example to read the content of a web page, and then step by step solve all the problems in progress.

1. Read the content of the Web page (HTTP/HTTPS)

Here's a simple example of how to access a page

1. /*  
2. * Created on 2003-12-14 by Liudong  
3. */  
4.   
5. ckage http.demo;   
6.   
7. import java.io.IOException;   
8. import org.apache.commons.httpclient.*;   
9. import org.apache.commons.httpclient.methods.*;  
10.   
11. /**  
12.  *The simplest HTTP client to demonstrate accessing a page by GET or POST 
13.   *@authorLiudong 
14. */  
15.   
16. public class SimpleClient {  
17. public static void main(String[] args) throws IOException {  
18.   HttpClient client = new HttpClient();   
19.       //Setting proxy server address and port  
20.   
21.       //client.getHostConfiguration().setProxy("proxy_host_addr",proxy_port);   
22.       //Using the GET method, if the server needs to connect via HTTPS, it only needs to replace http in the following URL with HTTPS  
23.          HttpMethod method=new GetMethod("http://java.sun.com");  
24.       //Using POST method  
25.       //HttpMethod method = new PostMethod("http://java.sun.com");  
26.       client.executeMethod(method);  
27.   
28.       //The status returned by the print server  
29.        System.out.println(method.getStatusLine());  
30.       //Print the returned information  
31.       System.out.println(method.getResponseBodyAsString());  
32.       //Release connection  
33.       method.releaseConnection();  
34.    }  
35. }  

 

In this example, we first create an instance of an HTTP client (HttpClient), then select the submission method as GET or POST, and finally execute the submission method on the HttpClient instance. Finally, we read the results from the server feedback from the selected submission method. This is the basic process of using HttpClient. In fact, a single line of code can be used to complete the entire request process, very simple!

2. Submit parameters to web pages by GET or POST

In fact, in the previous simplest example, we have already introduced how to request a page by GET or POST. This section is different from that in that it has more parameters to set when submitting. We know that if GET requests, then all parameters are placed directly behind the page URL and separated by question marks and page addresses. Each parameter is separated by & for example: http://java.sun.com/?name=liudong&mobile=123456 But when using the POST method, it's a little bit cumbersome. The example in this section demonstrates how to query the city where the mobile phone number is located. The code is as follows:

 

1. /*  
2. * Created on 2003-12-7 by Liudong  
3. */  
4.   
5. package com.zuidaima.http.demo;   
6. import java.io.IOException;   
7. import org.apache.commons.httpclient.*;   
8. import org.apache.commons.httpclient.methods.*;  
9.   
10. /**  
11.  *Submit parameter demonstration 
12.  *The program connects to a page for querying the location of a mobile phone number. 
13.  *For inquiring about the provinces and cities where the number section 1330227 is located 
14.  *@authorLiudong 
15.  */  
16.   
17. public class SimpleHttpClient {   
18.   
19.    public static void main(String[] args) throws IOException {  
20.       HttpClient client = new HttpClient();  
21.       client.getHostConfiguration().setHost( "www.imobile.com.cn" , 80, "http" );  
22.       method = getPostMethod();    //Submitting data using POST  
23.       client.executeMethod(method);   //The status returned by the print server  
24.       System.out.println(method.getStatusLine());   //Print the result page  
25.       Stringresponse=newString(method.getResponseBodyAsString().getBytes("8859_1"));  
26.   
27.       //Print the returned information  
28.       System.out.println(response);  
29.       method.releaseConnection();  
30.    }  
31.   
32.    /**  
33.  
34.     * Submit data using GET 
35.     *@return  
36.     */  
37.   
38.    privatestaticHttpMethodgetGetMethod(){  
39.       returnnewGetMethod("/simcard.php?simcard=1330227");  
40.    }  
41.   
42.   
43.   
44.     /**  
45.      * Submitting data using POST 
46.      *@return  
47.      */  
48.   
49.     private static HttpMethod getPostMethod(){  
50.       PostMethod post = new PostMethod( "/simcard.php" );  
51.       NameValuePair simcard = new NameValuePair( "simcard" , "1330227" );  
52.       post.setRequestBody( new NameValuePair[] { simcard});  
53.       return post;   
54.    }   
55. }  

In the example above, the page http://www.imobile.com.cn/simcard.php A parameter is simcard, which is the first seven digits of the mobile phone number. The server will return the province, city and other details of the submitted mobile phone number. The submission method of GET only needs to add parameter information after the URL, while POST needs to set the parameter name and its corresponding value through the NameValuePair class.

3. Processing page redirection

In JSP/Servlet programming, the response.sendRedirect method uses the redirection mechanism in HTTP protocol. It works with <jsp:forward... The difference is that the latter implements page Jump in the server, that is to say, the application container loads the content of the page to be jumped and returns it to the client; the former returns a status code whose possible values are shown in the table below, and the client reads the URL of the page to be jumped and reloads the new page. This is such a process, so we need to use the HttpMethod.getStatusCode() method to determine whether the return value is a value in the table to determine whether it needs to jump. If a page Jump is confirmed, the new address can be obtained by reading the location attribute in the HTTP header.

 

Status code	Constants corresponding to HttpServletResponse	Detailed description
301	SC_MOVED_PERMANENTLY	The page has been permanently moved to another new address
302	SC_MOVED_TEMPORARILY	Page temporarily moved to another new address
303	SC_SEE_OTHER	The address requested by the client must be accessed through another URL
307	SC_TEMPORARY_REDIRECT	Same as SC_MOVED_TEMPORARILY

The following code snippet demonstrates how to handle page redirection

1. client.executeMethod(post);  
2. System.out.println(post.getStatusLine().toString());  
3. post.releaseConnection();  
4. //Check for redirection  
5. int statuscode = post.getStatusCode();  
6. if ((statuscode == HttpStatus.SC_MOVED_TEMPORARILY) || (statuscode == HttpStatus.SC_MOVED_PERMANENTLY) || (statuscode ==HttpStatus.SC_SEE_OTHER) || (statuscode == HttpStatus.SC_TEMPORARY_REDIRECT)) {  
7. //Read the new URL address  
8.    Headerheader=post.getResponseHeader("location");  
9.    if (header!=null){  
10.       Stringnewuri=header.getValue();  
11.       if((newuri==null)||(newuri.equals("")))  
12.          newuri="/";  
13.          GetMethodredirect=newGetMethod(newuri);  
14.          client.executeMethod(redirect);  
15.          System.out.println("Redirect:"+redirect.getStatusLine().toString());  
16.          redirect.releaseConnection();  
17.    }else   
18.     System.out.println("Invalid redirect");  
19. }  

We can write two JSP pages ourselves, one of which is redirected to another page using the response.sendRedirect method. test The example above.

4. Simulate the input of username and password for login

This section should be said to be the most common problem encountered in HTTP client programming. The content of many websites is only visible to registered users. In this case, it is necessary to use the correct username and password to login successfully before browsing to the desired page. Because the HTTP protocol is stateless, that is, the validity of the connection is limited to the current request, the connection is closed after the request content is completed. In this case, the Cookie mechanism must be used to save the user's login information. Take JSP/Servlet as an example. When a browser requests a page of a JSP or Servlet, the application server returns a parameter named jsessionid (which varies from application server) whose value is a Cookie of a longer unique string, which is the session identifier currently accessing the site. Browsers should bring Cookie information such as jsessionid with them when visiting other pages of the site. Application servers can get corresponding session information by reading the session identity.

For websites requiring user login, user data is usually saved in the session of the server after successful login. When accessing other pages, the application server reads the session identifier corresponding to the current request according to the Cookie sent by the browser to obtain the corresponding session information. Then it can judge whether the user data exists in the session information, such as If it exists, it is allowed to visit the page. Otherwise, jump to the login page and ask the user to enter the account number and password for login. This is a common way to use JSP to develop websites to handle user logins.

In this way, for HTTP clients, if they want to access a protected page, they have to simulate what the browser does. First, they request the login page, then read the Cookie value; then they request the login page again and add every parameter required for the login page; finally, they request the final page. Of course, in addition to the first request, other requests need to be accompanied by cookie information so that the server can determine whether the current request has been verified. So much, but if you use httpclient, you don't even need to add a line of code. You just need to pass the login information to execute the login process, and then directly visit the desired page, which is no different from visiting a normal page, because the class HttpClient has already helped you to do everything you need to do. That's great! The following example implements such an access process.

1. /*  
2. * Created on 2003-12-7 by Liudong  
3. */  
4.   
5. package com.zuidaima.http.demo;   
6. import org.apache.commons.httpclient.*;  
7. import org.apache.commons.httpclient.cookie.*;  
8. import org.apache.commons.httpclient.methods.*;   
9.   
10. /**  
11.  * An example to demonstrate the login form 
12.  * @author Liudong  
13.  */  
14.   
15. public class FormLoginDemo {  
16.    static final String LOGON_SITE = "localhost" ;  
17.    static final int     LOGON_PORT = 8080;  
18.   
19.    public static void main(String[] args) throws Exception{  
20.       HttpClient client = new HttpClient();  
21.       client.getHostConfiguration().setHost(LOGON_SITE, LOGON_PORT);  
22.   
23.       //Simulated login page login.jsp - > main.jsp  
24.       PostMethod post = new PostMethod( "/main.jsp" );  
25.       NameValuePair name = new NameValuePair( "name" , "ld" );  
26.       NameValuePair pass = new NameValuePair( "password" , "ld" );  
27.       post.setRequestBody( new NameValuePair[]{name,pass});  
28.       int status = client.executeMethod(post);  
29.       System.out.println(post.getResponseBodyAsString());  
30.       post.releaseConnection();  
31.   
32.       //View cookie information  
33.       CookieSpec cookiespec = CookiePolicy.getDefaultSpec();  
34.       Cookie[] cookies = cookiespec.match(LOGON_SITE, LOGON_PORT, "/" , false , client.getState().getCookies());  
35.       if (cookies.length == 0) {  
36.          System.out.println( "None" );  
37.       } else {  
38.          for ( int i = 0; i < cookies.length; i++) {  
39.             System.out.println(cookies[i].toString());  
40.          }  
41.       }  
42.   
43.       //Accessing the required page main2.jsp  
44.       GetMethodget=newGetMethod("/main2.jsp");  
45.       client.executeMethod(get);  
46.       System.out.println(get.getResponseBodyAsString());  
47.       get.releaseConnection();  
48.    }  
49. }  

5. Submit XML format parameters

The parameters for submitting an XML format are simple, just a ContentType problem at the time of submission. The following example demonstrates the process of reading XML information from a file file and submitting it to the server, which can be used to test Web services.

1. package com.zuidaima.httpclient;  
2.   
3. import java.io.File;   
4. import java.io.FileInputStream;   
5. import org.apache.commons.httpclient.HttpClient;   
6. import org.apache.commons.httpclient.methods.EntityEnclosingMethod;   
7. import org.apache.commons.httpclient.methods.PostMethod;  
8.   
9.   
10. /**  
11.  *Examples for presenting data in XML format 
12. */  
13.   
14.   
15. public class PostXMLClient {  
16.   
17.    public static void main(String[] args) throws Exception {  
18.       File input = new File("test.xml");  
19.       PostMethod post = new PostMethod("http://localhost:8080/httpclient/xml.jsp");  
20.   
21.       //Set the content of the request to read directly from the file  
22.       post.setRequestBody( new FileInputStream(input));   
23.       if (input.length() < Integer.MAX_VALUE)  
24.          post.setRequestContentLength(input.length());  
25.       else  
26.          post.setRequestContentLength(EntityEnclosingMethod.CONTENT_LENGTH_CHUNKED);  
27.   
28.       //Specify the type of request content  
29.       post.setRequestHeader( "Content-type" , "text/xml; charset=GBK" );  
30.       HttpClient httpclient = new HttpClient();  
31.       int result = httpclient.executeMethod(post);  
32.       System.out.println( "Response status code: " + result);  
33.       System.out.println( "Response body: " );  
34.       System.out.println(post.getResponseBodyAsString());   
35.       post.releaseConnection();   
36.    }   
37. }  

6. Upload files over HTTP

httpclient uses a separate HttpMethod subclass to handle file upload. This class is MultipartPostMethod. This class has encapsulated the details of file upload. All we need to do is tell it the full path of file upload. The following code snippet demonstrates how to use this class.

1. MultipartPostMethod filePost = new MultipartPostMethod(targetURL);   
2. filePost.addParameter( "fileName" , targetFilePath);   
3. HttpClient client = new HttpClient();  
4.   
5. //Since the file to upload may be large, set the maximum connection timeout here.  
6. client.getHttpConnectionManager(). getParams().setConnectionTimeout(5000);   
7. int status = client.executeMethod(filePost);   

In the above code, targetFilePath is the path of the file to be uploaded.

7. Visit authentication-enabled pages

We often encounter pages that pop up when accessing a browser dialog asking for a user name and password. This user authentication method is different from the form-based user authentication we introduced earlier. This is the authentication strategy of HTTP. httpclient supports three authentication modes: basic, summary and NTLM authentication. Among them, basic authentication is the simplest, universal but also the most unsafe. Summary authentication is an authentication method added in HTTP 1.1, while NTLM is defined by Microsoft rather than a general specification. The latest version of NTLM is a safer way than summary authentication.

The following example is downloaded from the CVS server of httpclient, which simply demonstrates how to access an authenticated protected page:

1. package com.zuidaima.httpclient;  
2. import org.apache.commons.httpclient.HttpClient;   
3. import org.apache.commons.httpclient.UsernamePasswordCredentials;   
4. import org.apache.commons.httpclient.methods.GetMethod;   
5.   
6. public class BasicAuthenticationExample {   
7.   
8.    public BasicAuthenticationExample() {   
9.    }  
10.   
11.    public static void main(String[] args) throws Exception {  
12.       HttpClient client = new HttpClient();  
13.       client.getState().setCredentials( "www.verisign.com" , "realm" , new UsernamePasswordCredentials( "username" , "password") );  
14.   
15.       GetMethod get = new GetMethod( "https://www.verisign.com/products/index.html" );  
16.       get.setDoAuthentication( true );  
17.       int status = client.executeMethod( get );  
18.       System.out.println(status+ "\n" + get.getResponseBodyAsString());  
19.       get.releaseConnection();  
20.    }   
21. }  

8. Using httpclient in multithreaded mode

Multithreading accesses httpclient at the same time, such as downloading multiple files from one site at the same time. For the same HttpConnection, there can only be one thread access at the same time. To ensure that there is no conflict in the multi-threaded working environment, httpclient uses a class of multi-threaded connection manager: MultiThreaded HttpConnection Manager. To use this class is very simple, it only needs to be passed in when constructing an instance of HttpClient. The code is as follows:

MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();

HttpClient client = new HttpClient(connectionManager);

Later, you can access client instances.

Reference material:

httpclient Home Page: uuuuuuuuuuuuu http://jakarta.apache.org/commons/httpclient/
About how NTLM works: http://davenport.sourceforge.net/ntlm.html