Two questions

• spring boot RestTemplate unexpectedly reports null pointer exception after running for a period of time. You can locate that one HttpMessageConverter is null according to StackTrace;
• The content type returned by an interface is text/pain, but the returned result is always surrounded by double quotes, which leads to the failure of the interface caller's parsing. It can be located that a custom HttpMessageConverter intercepts the product type of text/plain by mistake, and adds double quotes at both ends of the string.

HttpMessageConverter function

HttpMessageConverter can turn different types of bodies into Java objects, or turn Java objects into bodies that meet the requirements. It plays a very important role in serialization and deserialization.

HttpMessageConverter matching rule

• The Http request header will contain Accept, which tells the server what kind of data to send back, such as Accept: application/json, text/javascript, */*; q=0.01; at the same time, it will specify the content type to tell the server what type of parameter data the body transmits this time. The server can convert the data type into the internal object of the server and extract the parameters;
• When the Server receives the request, it will judge whether it supports the parameter type transmitted from the client. If there is no support, the Server will return 406 (httpmediatype not supported exception);
• After processing the parameters and logic, the Server is ready to return according to the Accept required by the client. At this time, the Server will determine whether it supports the return of this type (MediaType). If not, the Server will return 406 (httpmediatype notacceptableexception);
• HttpMessageConverter is used by the server to determine whether it supports a certain MediaType;
• HttpMessageConverter is not only a list, but also a list. It is matched through the responsibility chain: traverse all httpmessageconverters in sequence, call its canRead() method, if it returns true, it means it can be processed. Once there is a HttpMessageConverter that can process a request's parameter MediaType, it uses the read() method of HttpMessageConverter to read the parameter Once the data is processed and about to be returned, traverse the HttpMessageConverter list with the same method, find the first HttpMessageConverter whose canWrite() returns true, and call its write() method to return it to the client.

HttpMessageConverter initialization sequence diagram

1. During ApplicationContextrefresh, HttpMessageConverter starts to initialize in three classes: RequestMappingHandlerAdapter and HttpMessageConverters autoconfiguration are simultaneous rows without interference, and HttpMessageConverters can only be initialized after the first two classes are initialized;
2. RequestMappingHandlerAdapter initializes the default HttpMessageConverter list
   • Call the configureMessageConverters method of all WebMvcConfigurer type custom Configuration classes (@ Configuration) to initialize the HttpMessageConverter list;
   • If there is no custom WebMvcConfigurer configuration, call addDefaultHttpMessageConverters method to initialize the HttpMessageConverter list. The default HttpMessageConverter list is to determine whether a certain HttpMessageConverter needs to be added to the default list according to whether or not a specific class is loaded in ClassLoader, and sort it at the end, just xml type At the end of the current HttpMessageConverter list;
   • Call the extendMessageConverters method of all WebMvcConfigurer type custom configuration classes to extend the HttpMessageConverter list, and add it directly to the end of the list
3. HttpMessageConvertersAutoConfiguration class initializes all components of HttpMessageConverter type (@ contractor / @ bean, etc.) into a HttpMessageConverter list provided by context;
4. When HttpMessageConverters are initialized, merge the 2 and 3 lists. If the context provided and the default list are duplicate but the object is not the same, the HttpMessageConverter provided by the context and the HttpMessageConverter in the default list will be placed adjacent to each other, and the context provided will be placed first; all the context provided and not in the default list HttpMessageConverter in is placed at the top of the whole merge list. The order of HttpMessageConverter provided by the context is specified by @ Order(value=1) annotation on the class. The smaller the value, the higher the priority.

Custom HttpMessageConverter

package com.enmo.dbaas.common.config.feigninterceptor;

import org.springframework.core.annotation.Order;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.HttpOutputMessage;
import org.springframework.http.MediaType;
import org.springframework.http.converter.AbstractHttpMessageConverter;
import org.springframework.http.converter.GenericHttpMessageConverter;
import org.springframework.http.converter.HttpMessageNotReadableException;
import org.springframework.http.converter.HttpMessageNotWritableException;
import org.springframework.lang.Nullable;
import org.springframework.stereotype.Component;

import javax.activation.UnsupportedDataTypeException;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Type;

/**
 * Created by IntelliJ IDEA
 *
 * @author chenlei
 * @date 2020/1/11
 * @time 17:44
 * @desc AHttpMessageConverter
 */
@Component
@Order(1)
public class AHttpMessageConverter extends AbstractHttpMessageConverter<Object> implements GenericHttpMessageConverter<Object> {
    @Override
    protected boolean supports(Class<?> clazz) {
        return true;
    }

    @Override
    protected Object readInternal(Class<?> clazz, HttpInputMessage inputMessage) throws IOException, HttpMessageNotReadableException {
        return null;
    }

    @Override
    protected void writeInternal(Object o, HttpOutputMessage outputMessage) throws IOException, HttpMessageNotWritableException {
        outputMessage.getBody().write("{}".getBytes());
        outputMessage.getBody().flush();
    }

    @Override
    public boolean canRead(Type type, @Nullable Class<?> contextClass, @Nullable MediaType mediaType) {
        return true;
    }

    @Override
    public Object read(Type type, @Nullable Class<?> contextClass, HttpInputMessage inputMessage) throws IOException, HttpMessageNotReadableException {
        InputStream in = inputMessage.getBody();
        if (String.class.getTypeName().equals(type.getTypeName())) {
            byte[] bytes = new byte[65536];
            int offset = 0;

            while(true) {
                int readCount = in.read(bytes, offset, bytes.length - offset);
                if(readCount == -1) {
                    return new String(bytes, "UTF-8");
                }

                offset += readCount;
                if(offset == bytes.length) {
                    byte[] newBytes = new byte[bytes.length * 3 / 2];
                    System.arraycopy(bytes, 0, newBytes, 0, bytes.length);
                    bytes = newBytes;
                }
            }
        }
        throw new UnsupportedDataTypeException(type.getClass().getTypeName());
    }

    @Override
    public boolean canWrite(@Nullable Type type, Class<?> clazz, @Nullable MediaType mediaType) {
        return true;
    }

    @Override
    public void write(Object o, @Nullable Type type, @Nullable MediaType contentType, HttpOutputMessage outputMessage) throws IOException, HttpMessageNotWritableException {
        super.write(o, contentType, outputMessage);
    }
}

Define an AHttpMessageConverter, canRead() and canWrite() to directly return true, and set the @ Order value to 1, because the context HttpMessageConverter list using Component annotation is added in front of HttpMessageConverter, and the default HttpMessageConverter value of spring is Integer.MAX_VALUE, so AHttpMessageConverter will match all Content type and Accept.

The read() method in the class only supports the parameters of String type, and all parameters of other types throw unsupportateddatatypeexception;
The write() method in the class returns the string '{}'.

Suppose you write a method like this in the Controller:

@PostMapping("install/invoke/test")
public ResultData invokeTest(@RequestHeader String a,
                         @RequestParam String b,
                         @RequestParam String c,
                         @RequestBody JSONObject d) {
    log.info("{}, {}, {} ,{}", a, b, c, d);
    return new JSONObject().fluentPut("key", "value");
}

Will report wrong:

org.springframework.http.converter.HttpMessageNotReadableException: I/O error while reading input message; nested exception is javax.activation.UnsupportedDataTypeException: java.lang.Class
	at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:216)
	at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:157)
	at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:130)
	at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:126)
	at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:167)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.activation.UnsupportedDataTypeException: java.lang.Class
	at com.enmo.dbaas.common.config.feigninterceptor.AHttpMessageConverter.read(AHttpMessageConverter.java:72)
	at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:204)
	... 62 common frames omitted

If you change the JSONObject to String:

@PostMapping("install/invoke/test")
public ResultData invokeTest(@RequestHeader String a,
                         @RequestParam String b,
                         @RequestParam String c,
                         @RequestBody String d) {
    log.info("{}, {}, {} ,{}", a, b, c, d);
    return new JSONObject().fluentPut("key", "value");
}

Will return:

{}

Instead of:

{
	"key": "value"
}

Note that both Accept and content type use AHttpMessageConverter to serialize and deserialize.

Solve the problem

RestTemplate NPE

spring boot RestTemplate unexpectedly reports null pointer exception after running for a period of time. You can locate that there is an HttpMessageConverter empty according to StackTrace

It will be empty after a period of time. There must be a place in the running process to change the value of an index of HttpMessageConverter list to null. It is not to report an error as soon as it is started, nor to call a specific interface. After that, every request will report an error.

I just found this article: Set StringHttpMessageConverter to handle messageConverters of restTemplate

It is mentioned in the article that the character set is set to UTF-8 manually every time the RestTemplate is called, and NPE is occasionally sent online, which is very similar to our setting:

restTemplate.getMessageConverters()
        .add(0, new StringHttpMessageConverter(StandardCharsets.UTF_8));

We know that the list < HttpMessageConverter > in the HttpMessageConverters class is just a simple unmodifiable list < HttpMessageConverter <? > > which does not support high concurrency. In this way, either the list < HttpMessageConverter > will be inflated infinitely, or the null pointer will be caused, because the principle of List.add(int index, Object o) is to move all elements of the list to an index first, and then the index In the case of high concurrency, another thread may have started to traverse. When the first HttpMessageConverter is null, null pointer will be reported, which is the accidental reason;

Then why is it possible that after one occurrence, all the subsequent requests will report null pointers? Especially when there are multiple scheduled tasks running at the same time in the morning, the probability of "permanent NPE" is very high. This is because there are just two threads setting HttpMessageConverter with index 0 at the same time. One thread has just moved all elements back one bit, and it has not yet time to assign a value to the position with index 0. The other thread has started to move all elements back again, causing the element with index 1 to be permanently empty, and it can no longer be saved.

The solution in the article is similar to the single instance mode, but it can't completely avoid the high concurrency problem. You still need to add synchronization blocks and use duoble check. This way is not elegant. Because the spring bean is a single instance by default, we can completely configure it at the time of initialization, and then directly inject it for use, without secondary configuration:

@Configuration
public class HttpConfiguration {
    @Bean
    RestTemplate restTemplate() {
        RestTemplate restTemplate = new RestTemplate();
        List<HttpMessageConverter<?>> messageConverters = restTemplate.getMessageConverters();
        messageConverters.removeIf(converter -> converter instanceof StringHttpMessageConverter);
        messageConverters.add(0, new StringHttpMessageConverter(StandardCharsets.UTF_8));
        return restTemplate;
    }
}

Text / painquoted

The content type returned by an interface is text/pain, but the returned result is always surrounded by double quotes, which leads to the failure of the interface caller's parsing. It can be located that a custom HttpMessageConverter intercepts the product type of text/plain by mistake, and adds double quotes at both ends of the string.

After understanding the above principle, we can guess that it is caused by wrong HttpMessageConverter serialization. When no custom HttpMessageConverter is configured, everything is normal. We only add a custom FastJsonHttpMessageConverter:

package com.enmo.dbaas.common.config;

import com.alibaba.fastjson.serializer.SerializerFeature;
import com.alibaba.fastjson.support.config.FastJsonConfig;
import com.alibaba.fastjson.support.spring.FastJsonHttpMessageConverter;
import org.springframework.core.annotation.Order;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Component;

import java.util.ArrayList;
import java.util.List;

@Component
public class FastJsonHttpMessageConverterEx extends FastJsonHttpMessageConverter {
    public FastJsonHttpMessageConverterEx() {
        FastJsonConfig fastJsonConfig = new FastJsonConfig();
        fastJsonConfig.setDateFormat("yyyy-MM-dd HH:mm:ss");    // Custom time format
        fastJsonConfig.setSerializerFeatures(SerializerFeature.DisableCircularReferenceDetect, SerializerFeature.WriteMapNullValue); // Normal conversion null value
        List<MediaType> supportedMediaTypes = new ArrayList<>();
        supportedMediaTypes.add(MediaType.APPLICATION_JSON);
        supportedMediaTypes.add(MediaType.APPLICATION_JSON_UTF8);
        supportedMediaTypes.add(MediaType.APPLICATION_ATOM_XML);
        supportedMediaTypes.add(MediaType.APPLICATION_FORM_URLENCODED);
        supportedMediaTypes.add(MediaType.APPLICATION_OCTET_STREAM);
        supportedMediaTypes.add(MediaType.APPLICATION_PDF);
        supportedMediaTypes.add(MediaType.APPLICATION_RSS_XML);
        supportedMediaTypes.add(MediaType.APPLICATION_XHTML_XML);
        supportedMediaTypes.add(MediaType.APPLICATION_XML);
        supportedMediaTypes.add(MediaType.IMAGE_GIF);
        supportedMediaTypes.add(MediaType.IMAGE_JPEG);
        supportedMediaTypes.add(MediaType.IMAGE_PNG);
        supportedMediaTypes.add(MediaType.TEXT_EVENT_STREAM);
        supportedMediaTypes.add(MediaType.TEXT_HTML);
        supportedMediaTypes.add(MediaType.TEXT_MARKDOWN);
        supportedMediaTypes.add(MediaType.TEXT_XML);
        supportedMediaTypes.add(MediaType.TEXT_PLAIN);
        this.setSupportedMediaTypes(supportedMediaTypes);
        this.setFastJsonConfig(fastJsonConfig);
    }
}

This is provided by the context and will be placed in the first of the HttpMessageConverter list, with the highest priority.
When combining Fastjson to serialize strings, quotation marks will be added:

Therefore, it must be the pot of FastJsonHttpMessageConverterEx. Sure enough, the canRead() method and canWrite() method of FastJsonHttpMessageConverter are judged according to the list < MediaType > supportedmediatypes: as long as the MediaType defined by list < MediaType > supportedmediatypes is serialized and deserialized by FastJsonHttpMessageConverterEx, we have Fixed FastJsonHttpMessageConverterEx plus supportedmediatypes. Add (MediaType. Text ﹣ plan);, it's strange that there is no error.

Delete the line, solve the problem, and consider whether FastJsonHttpMessageConverterEx should handle so many types. Although the system works well at present, there is no guarantee that there will be problems after the new business is added. At present, we only have application / JSON; plain / text; application / object stream and other types. If an image/jpeg appears one day, it will not be even There is an error because Fastjson may not be able to handle it, but it is forced to handle it, or it may be forced to handle it, and the result format is not correct.