Hystrix: Service fuse
Problems faced by distributed systems
There are dozens of dependencies for applications in complex distributed architecture, and each dependency will inevitably fail at some time!
1. Service avalanche
When calling between multiple microservices, suppose microservice A calls microservice B and microservice C, and microservice B and microservice C call other microservices, which is the so-called "fan out". If the call response time of A microservice on the fan out link is too long or unavailable, the call to microservice A will occupy more and more system resources, resulting in system crash, The so-called "avalanche effect".
For high traffic applications, a single back-end dependency may cause all resources on all servers to saturate in tens of seconds. Worse than failure, these applications may also lead to increased delays between services, tight backup queues, threads and other system resources, resulting in more cascading failures of the whole system. These all mean that failures and delays need to be isolated and managed to achieve the failure of a single dependency without affecting the operation of the whole application or system.
2. What is Hystrix?
Hystrix is an open source library used to deal with the delay and fault tolerance of distributed systems. In distributed systems, many dependencies inevitably fail to call, such as timeout, exception, etc. hystrix can ensure that when a dependency fails, it will not lead to the failure of the whole system service, avoid cascading failures, and improve the elasticity of distributed systems.
"Circuit breaker" itself is a kind of switching device. When a service unit fails, it returns a service expected and processable alternative response (FallBack) to the caller through the fault monitoring of the circuit breaker (similar to a blown fuse), rather than waiting for a long time or throwing an exception that cannot be handled by the calling method, This can ensure that the thread of the service caller will not be occupied unnecessarily for a long time, so as to avoid the spread and even avalanche of faults in the distributed system.
3. Service fuse
What is service?
Circuit breaker mechanism is a microservice link protection mechanism to win avalanche effect.
When a microservice of the fan out link is unavailable or the response time is too long, the service will be degraded, which will fuse the call of the microservice of the node and quickly return the wrong response information. After detecting that the microservice call response of the node is normal, restore the call link. In the spring cloud framework, the fuse mechanism is implemented through hystrix. Hystrix will monitor the status of calls between microservices. When the failed call reaches a certain threshold, the default is 20 calls in 5 seconds, and the fuse mechanism will be started. The annotation of the fuse mechanism is: @ HystrixCommand.
The following problems can be solved by the service provider:
When the dependent object is unstable, it can achieve the purpose of rapid failure;
After a fast failure, it can dynamically test whether the dependent object is restored according to a certain algorithm.
Introductory case
Create a new springcloud provider Dept hystrix-8001 module and copy the pom.xml, resource and Java code in springcloud provider Dept – 8001 for initialization and adjustment.
Import hystrix dependencies
<!--Import Hystrix rely on-->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-hystrix</artifactId>
<version>1.4.6.RELEASE</version>
</dependency>
Adjust yml profile
server:
port: 8001
# mybatis configuration
mybatis:
# pojo package under spring cloud API module
type-aliases-package: com.haust.springcloud.pojo
# Class path of mybatis-config.xml core configuration file under this module
config-location: classpath:mybatis/mybatis-config.xml
# mapper configuration file classpath under this module
mapper-locations: classpath:mybatis/mapper/*.xml
# spring configuration
spring:
application:
#Project name
name: springcloud-provider-dept
datasource:
# Druid data source
type: com.alibaba.druid.pool.DruidDataSource
driver-class-name: com.mysql.jdbc.Driver
url: jdbc:mysql://localhost:3306/db01?useUnicode=true&characterEncoding=utf-8
username: root
password: root
# Eureka configuration: configure the service registry address
eureka:
client:
service-url:
# Registration center address 7001-7003
defaultZone: http://eureka7001.com:7001/eureka/,http://eureka7002.com:7002/eureka/,http://eureka7003.com:7003/eureka/
instance:
instance-id: springcloud-provider-dept-hystrix-8001 #Modify the default description information on Eureka
prefer-ip-address: true #After changing to true, the default display is the ip address instead of localhost
#info configuration
info:
app.name: haust-springcloud #Name of the project
company.name: com.haust #Company name
prefer-ip-address: false:
prefer-ip-address: true:
Modify controller
@RestController
public class DeptController {
@Autowired
private DeptService deptService;
/**
* Query department information according to id
* If an exception occurs in the query based on id, follow the alternative code of hystrixGet
* @param id
* @return
*/
@HystrixCommand(fallbackMethod = "hystrixGet")
@RequestMapping("/dept/get/{id}")//Query by id
public Dept get(@PathVariable("id") Long id){
Dept dept = deptService.queryById(id);
if (dept==null){
throw new RuntimeException("this id=>"+id+",The user does not exist or the information cannot be found~");
}
return dept;
}
/**
* Query alternatives by id (fuse)
* @param id
* @return
*/
public Dept hystrixGet(@PathVariable("id") Long id){
return new Dept().setDeptno(id)
.setDname("this id=>"+id+",No corresponding information,null---@Hystrix~")
.setDb_source("stay MySQL This database does not exist in");
}
}
Add the support annotation @ enablercircuitbreaker to the main startup class
@SpringBootApplication
@EnableEurekaClient // The startup class of EnableEurekaClient client automatically registers the service with the registry after the service is started
@EnableDiscoveryClient // Service discovery~
@EnableCircuitBreaker // Add a supporting comment for the fuse
public class HystrixDeptProvider_8001 {
public static void main(String[] args) {
SpringApplication.run(HystrixDeptProvider_8001.class,args);
}
}
Test:
After use, when accessing a nonexistent id, the foreground page displays the following data:
If the springcloud provider Dept – 8001 module that is not applicable accesses the same address, the following conditions will occur:
Therefore, it is necessary to use fuse in order to avoid errors in the whole application or web page due to exceptions or errors in the background of a micro service
4. Service degradation
What is service degradation?
Service degradation means that when the server pressure increases sharply, some services and pages are not processed strategically or handled in a simple way according to the actual business conditions and traffic, so as to release server resources to ensure the normal or efficient operation of core business. To put it bluntly, it is to give system resources to high priority services as much as possible.
Resources are limited and requests are unlimited. If the service is not degraded during the concurrency peak period, on the one hand, it will certainly affect the performance of the overall service. If it is serious, it may lead to downtime and unavailability of some important services. Therefore, in order to ensure the availability of core function services, some services must be degraded during peak periods. For example, when double 11 is active, all services unrelated to the transaction are downgraded, such as viewing ant deep forest, viewing historical orders, etc.
What are the main scenarios for service degradation? When the overall load of the whole microservice architecture exceeds the preset upper threshold or the upcoming traffic is expected to exceed the preset threshold, some unimportant or non urgent services or tasks can be delayed or suspended in order to ensure the normal operation of important or basic services.
The degradation method can be based on the business, and the service can be delayed. For example, the delay will increase the points for the user, but it will be put into a cache and executed after the service is stable; Or close the service within the granularity, such as the recommendation of relevant articles.
It can be seen from the above figure that when the traffic of service A increases sharply and the traffic of B and C is small at A certain time, in order to alleviate the pressure of service A, B and C need to temporarily close some service functions to undertake some services of A, so as to share the pressure for A, which is called service degradation.
Issues to be considered for service degradation
1) Which services are core services and which are non core services
2) Which services can support degradation, which services cannot support degradation, what is the degradation strategy
3) In addition to service degradation, is there a more complex business release scenario and what is the strategy?
Automatic degradation classification
1) Timeout degradation: it mainly configures the timeout time and timeout retry times and mechanism, and uses asynchronous mechanism to detect the reply
2) Degradation of failure times: it mainly refers to some unstable APIs. When the number of failed calls reaches a certain threshold, it will be degraded automatically. Similarly, asynchronous mechanism should be used to detect the reply
3) Failure degradation: for example, if the remote service to be called hangs (network failure, DNS failure, http service returns an error status code, rpc service throws an exception), it can be directly degraded. The processing schemes after degradation include: default value (for example, if the inventory service is suspended, it will return to the default stock), bottom data (for example, if the advertisement is suspended, it will return some static pages prepared in advance), cache (some cache data previously temporarily stored)
4) Current limit degradation: when second killing or snapping up some restricted goods, the system may crash due to too much traffic. At this time, current limit will be used to limit the traffic. When the current limit threshold is reached, subsequent requests will be degraded; The processing scheme after degradation can be: queuing page (divert the user to the queuing page and try again later), no goods (directly inform the user that there is no goods), error page (if the activity is too hot, try again later).
Introductory case
Create a new degraded configuration class DeptClientServiceFallBackFactory.java in the service package under the springcloud API module
@Component
public class DeptClientServiceFallBackFactory implements FallbackFactory {
@Override
public DeptClientService create(Throwable cause) {
return new DeptClientService() {
@Override
public Dept queryById(Long id) {
return new Dept()
.setDeptno(id)
.setDname("id=>" + id + "There is no corresponding information. The client provides degraded information. The service has been shut down")
.setDb_source("no data~");
}
@Override
public List<Dept> queryAll() {
return null;
}
@Override
public Boolean addDept(Dept dept) {
return false;
}
};
}
}
Specify the degraded configuration class DeptClientServiceFallBackFactory in DeptClientService
@Component //Register in spring container
//@FeignClient: Micro service client annotation, value: specify the name of the micro service, so that the Feign client can directly find the corresponding micro service
@FeignClient(value = "SPRINGCLOUD-PROVIDER-DEPT",fallbackFactory = DeptClientServiceFallBackFactory.class)//fallbackFactory specifies the degraded configuration class
public interface DeptClientService {
@GetMapping("/dept/get/{id}")
public Dept queryById(@PathVariable("id") Long id);
@GetMapping("/dept/list")
public List<Dept> queryAll();
@GetMapping("/dept/add")
public Boolean addDept(Dept dept);
}
Enable demotion in the springcloud consumer dept feign module:
server:
port: 80
# Eureka configuration
eureka:
client:
register-with-eureka: false # Do not register yourself with Eureka
service-url: # Access one of the three registries at random
defaultZone: http://eureka7001.com:7001/eureka/,http://eureka7002.com:7002/eureka/,http://eureka7003.com:7003/eureka/
# Open degraded feign.hystrix
feign:
hystrix:
enabled: true
5. Difference between service fusing and degradation
Service fusing - > server: a service timeout or exception causes fusing ~, which is similar to a fuse (Self Fusing)
Service degradation - > client: considering the overall website request load, when a service is blown or closed, the service will no longer be called. At this time, on the client, we can prepare a FallBackFactory and return a default value (default value). It will lead to a decline in the overall service, but at least it can be used, which is better than hanging up directly.
The triggering reasons are different. Service fusing is generally caused by a service (downstream service) fault, and service degradation is generally considered from the overall load; The levels of management objectives are different. Fusing is actually a framework level process, and each micro service needs to be (without hierarchy), while degradation generally needs to have hierarchy for the business (for example, degradation generally starts from the most peripheral services)
The implementation method is different. Service degradation is code intrusive (completed by the controller and / or automatic degradation), and fusing is generally called self fusing.
Fusing, degradation, current limiting:
Flow limit: limit concurrent request access. If the threshold is exceeded, it will be rejected;
Degradation: services are prioritized, sacrificing non core services (unavailable) to ensure the stability of core services; Considering the overall load;
Fusing: the dependent downstream service failure triggers fusing to avoid causing the system crash; Automatic system execution and recovery
6. Dashboard flow monitoring
Create a new springcloud consumer hystrix dashboard module
Add dependency
<!--Hystrix rely on-->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-hystrix</artifactId>
<version>1.4.6.RELEASE</version>
</dependency>
<!--dashboard rely on-->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-hystrix-dashboard</artifactId>
<version>1.4.6.RELEASE</version>
</dependency>
<!--Ribbon-->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-ribbon</artifactId>
<version>1.4.6.RELEASE</version>
</dependency>
<!--Eureka-->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-eureka</artifactId>
<version>1.4.6.RELEASE</version>
</dependency>
<!--Entity class+web-->
<dependency>
<groupId>com.haust</groupId>
<artifactId>springcloud-api</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!--Hot deployment-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
</dependency>
Main startup class
@SpringBootApplication
// Open Dashboard
@EnableHystrixDashboard
public class DeptConsumerDashboard_9001 {
public static void main(String[] args) {
SpringApplication.run(DeptConsumerDashboard_9001.class,args);
}
}
to springcloud-provider-dept-hystrix-8001 Add the following code to the main startup class under the module,Add monitoring
@SpringBootApplication
@EnableEurekaClient //The startup class of EnableEurekaClient client automatically registers the service with the registry after the service is started
public class DeptProvider_8001 {
public static void main(String[] args) {
SpringApplication.run(DeptProvider_8001.class,args);
}
//Add a Servlet
@Bean
public ServletRegistrationBean hystrixMetricsStreamServlet(){
ServletRegistrationBean registrationBean = new ServletRegistrationBean(new HystrixMetricsStreamServlet());
//Accessing this page is the monitoring page
registrationBean.addUrlMappings("/actuator/hystrix.stream");
return registrationBean;
}
}
visit: http://localhost:9001/hystrix
Enter the monitoring page:
reference material
Official website information: https://github.com/Netflix/Hystrix/wiki