ABSTRACT: Origin http://www.iocoder.cn/Eureka/instance-registry-evict/ "Taro Source Code" Welcome to reprint, keep the abstract, thank you!

This paper is mainly based on Eureka 1.8.X version.

• 1. overview
• 2. Why do we need to expire?
• 3. EvictionTask
• 4. Outdated Logic
• 666. eggs

1. overview

This paper mainly shares Eureka-Server's overdue renewal lease.

Recommend Spring Cloud books:

• Please support the original version. Downloading piracy is equivalent to actively writing low-level BUG.
• Programmed ape DD—— Spring Cloud Microservice Actual Warfare
• Zhou Li—— Spring Cloud and Docker Microsoft Architecture in Practice
• Two books are bought together and packaged in Beijing and East China.

Recommend Spring Cloud videos:

• Java Microservice Practice - Spring Boot
• Java Microservice Practice - Spring Cloud
• Java Micro Service Practice - Spring Boot / Spring Cloud

2. Why do we need to expire?

Normally, when an application instance is offline, it initiates an offline request to Eureka-Server. However, in practice, application instances may crash abnormally, or network anomalies and other reasons, resulting in offline requests can not be successfully submitted.

In this case, we extend the lease through Eureka-Client heartbeat, and cooperate with Eureka-Server to clean up the overtime lease to solve the above anomalies.

3. EvictionTask

com.netflix.eureka.registry.AbstractInstanceRegistry.EvictionTask, clean up lease expiration tasks. When Eureka-Server starts up, it initializes EvictionTask to execute at a fixed time. The implementation code is as follows:

// AbstractInstanceRegistry.java
/**
* Clean up the expired lease
*/
private final AtomicReference<EvictionTask> evictionTaskRef = new AtomicReference<EvictionTask>();

protected void postInit() {
   // .... Eliminate irrelevant code

   // Initialize Clean-up Rent Expiration Task
   if (evictionTaskRef.get() != null) {
       evictionTaskRef.get().cancel();
   }
   evictionTaskRef.set(new EvictionTask());
   evictionTimer.schedule(evictionTaskRef.get(),
           serverConfig.getEvictionIntervalTimerInMs(),
           serverConfig.getEvictionIntervalTimerInMs());
}

• Configure eureka. eviction IntervalTimerInMs to clear out the frequency of expired lease tasks in milliseconds. By default, 60,000 milliseconds.

• The EvictionTask implementation code is as follows:

  class EvictionTask extends TimerTask {
  
     @Override
     public void run() {
         try {
             // Get the number of milliseconds of compensation time
             long compensationTimeMs = getCompensationTimeMs();
             logger.info("Running the evict task with compensationTime {}ms", compensationTimeMs);
             // Cleaning up the logic of expired leases
             evict(compensationTimeMs);
         } catch (Throwable e) {
             logger.error("Could not run the evict task", e);
         }
     }
  
  }
  

  • Call the # compensationTimeMs() method to get the compensation time in milliseconds. Calculating formula = current time - last task execution time - task execution frequency. Why do we need to compensate for the milliseconds? "4. Outdated Logic" Lease isisExpired (additionalLeaseMs) Method Announced. # The compensationTimeMs() implementation code is as follows:

    /**
    * Final Task Execution Time
    */
    private final AtomicLong lastExecutionNanosRef = new AtomicLong(0L);
    
    long getCompensationTimeMs() {
        long currNanos = getCurrentTimeNano();
        long lastNanos = lastExecutionNanosRef.getAndSet(currNanos);
        if (lastNanos == 0L) {
            return 0L;
        }
        long elapsedMs = TimeUnit.NANOSECONDS.toMillis(currNanos - lastNanos);
        long compensationTime = elapsedMs - serverConfig.getEvictionIntervalTimerInMs();
        return compensationTime <= 0L ? 0L : compensationTime;
    }
    

    • Due to JVM GC, or clock skew, timer execution is actually slightly delayed than expected. The author runs under low load, about 10 ms.

      compute a compensation time defined as the actual time this task was executed since the prev iteration, vs the configured amount of time for execution. This is useful for cases where changes in time (due to clock skew or gc for example) causes the actual eviction task to execute later than the desired time according to the configured cycle.

  • Call the # evict(compensationTime) method to execute the cleanup expired lease logic, in the "4. Outdated Logic" Detailed analysis.

4. Outdated Logic

Call the # evict(compensationTime) method to execute the logic of cleaning up expired leases. The implementation code is as follows:

  1: public void evict(long additionalLeaseMs) {
  2:     logger.debug("Running the evict task");
  3: 
  4:     if (!isLeaseExpirationEnabled()) {
  5:         logger.debug("DS: lease expiration is currently disabled.");
  6:         return;
  7:     }
  8: 
  9:     // Obtain all expired leases
 10:     // We collect first all expired items, to evict them in random order. For large eviction sets,
 11:     // if we do not that, we might wipe out whole apps before self preservation kicks in. By randomizing it,
 12:     // the impact should be evenly distributed across all applications.
 13:     List<Lease<InstanceInfo>> expiredLeases = new ArrayList<>();
 14:     for (Entry<String, Map<String, Lease<InstanceInfo>>> groupEntry : registry.entrySet()) {
 15:         Map<String, Lease<InstanceInfo>> leaseMap = groupEntry.getValue();
 16:         if (leaseMap != null) {
 17:             for (Entry<String, Lease<InstanceInfo>> leaseEntry : leaseMap.entrySet()) {
 18:                 Lease<InstanceInfo> lease = leaseEntry.getValue();
 19:                 if (lease.isExpired(additionalLeaseMs) && lease.getHolder() != null) { // Be overdue
 20:                     expiredLeases.add(lease);
 21:                 }
 22:             }
 23:         }
 24:     }
 25: 
 26:     // Calculate the maximum number of clearance leases allowed
 27:     // To compensate for GC pauses or drifting local time, we need to use current registry size as a base for
 28:     // triggering self-preservation. Without that we would wipe out full registry.
 29:     int registrySize = (int) getLocalRegistrySize();
 30:     int registrySizeThreshold = (int) (registrySize * serverConfig.getRenewalPercentThreshold());
 31:     int evictionLimit = registrySize - registrySizeThreshold;
 32: 
 33:     // Calculate the number of clean-up leases
 34:     int toEvict = Math.min(expiredLeases.size(), evictionLimit);
 35:     if (toEvict > 0) {
 36:         logger.info("Evicting {} items (expired={}, evictionLimit={})", toEvict, expiredLeases.size(), evictionLimit);
 37: 
 38:         // Expire one by one
 39:         Random random = new Random(System.currentTimeMillis());
 40:         for (int i = 0; i < toEvict; i++) {
 41:             // Pick a random item (Knuth shuffle algorithm)
 42:             int next = i + random.nextInt(expiredLeases.size() - i);
 43:             Collections.swap(expiredLeases, i, next);
 44:             Lease<InstanceInfo> lease = expiredLeases.get(i);
 45: 
 46:             String appName = lease.getHolder().getAppName();
 47:             String id = lease.getHolder().getId();
 48:             EXPIRED.increment();
 49:             logger.warn("DS: Registry: expired lease for {}/{}", appName, id);
 50:             internalCancel(appName, id, false);
 51:         }
 52:     }
 53: }

• Lines 3 to 7: Judging that the logic of clearing up expired leases is allowed to be implemented, which is mainly related to the self-protection mechanism. Eureka Source Parsing: Application Case Registration Discovery (4) Self-protection Mechanism Detailed analysis.

• Lines 9 to 24: Get all outdated lease sets.

  • Line 19: Call the Lease isisExpired (additionalLeaseMs) method to determine if the lease is expired. The implementation code is as follows:

    // Lease.java
    public boolean isExpired(long additionalLeaseMs) {
       return (evictionTimestamp > 0 || System.currentTimeMillis() > (lastUpdateTimestamp + duration + additionalLeaseMs));
    }
    
    public void renew() {
       lastUpdateTimestamp = System.currentTimeMillis() + duration;
    }
    

    • (iv) Note: Without considering the additionalLeaseMs parameter, the lease expiration time is one more duration than expected because the #renew() method incorrectly sets lastUpdate Timestamp = System. CurrtTimeMillis () + duration, and the correct setting should be lastUpdate Timestamp = System. CurrtTimeMillis ().

      Note that due to renew() doing the 'wrong" thing and setting lastUpdateTimestamp to +duration more than what it should be, the expiry will actually be 2 * duration. This is a minor bug and should only affect instances that ungracefully shutdown. Due to possible wide ranging impact to existing usage, this will not be fixed.

    • TODO[0023]: additionalLeaseMs

• Lines 26 to 34: Calculate the maximum number of clearance leases allowed, and then calculate the number of clearance leases allowed.

  • (iv) Note: Even if Eureka-Server closes the self-protection mechanism, if renewal PercentThreshold = 0.85 default configuration is used, the result will be gradual expiration of batches. For instance:

    // Suppose 20 leases, 10 of which are expired.
    
    // The first round of implementation begins
    int registrySize = 20;
    int registrySizeThreshold = (int) (20 * 0.85) = 17;
    int evictionLimit = 20 - 17 = 3;
    int toEvict = Math.min(10, 3) = 3;
    // The first round of implementation ended with 17 remaining leases, of which 7 were overdue.
    
    // Starting of the second round of implementation
    int registrySize = 17;
    int registrySizeThreshold = (int) (17 * 0.85) = 14;
    int evictionLimit = 17 - 14 = 3;
    int toEvict = Math.min(7, 3) = 3;
    // The second round of implementation ended with the remaining 14 leases, four of which were overdue.
    
    // The third round of implementation begins
    int registrySize = 14;
    int registrySizeThreshold = (int) (14 * 0.85) = 11;
    int evictionLimit = 14 - 11 = 3;
    int toEvict = Math.min(4, 3) = 3;
    // The third round of implementation ended with 11 remaining leases, one of which expired.
    
    // Starting of the fourth round of implementation
    int registrySize = 11;
    int registrySizeThreshold = (int) (11 * 0.85) = 9;
    int evictionLimit = 11 - 9 = 2;
    int toEvict = Math.min(1, 2) = 1;
    // The fourth round of implementation ended with the remaining 10 leases, of which 0 had expired. End.
    

    • Conclusion: The difference between self-protection and non-self-protection lies in the implementation of the logic of clearing out expired leases. If you want to close the gradual expiration of batches, set renewalPercentThreshold = 0.

  • Due to JVM GC, or local time differences, the thresholds of self-protection mechanisms, expectedNumberOfRenews PerMin and NumberOfRenews PerMinThreshold, may not be correct enough. In this relatively "dangerous" operation, self-protection thresholds are recalculated in the expiration period.

• Lines 35 to 51: Randomly clean up expired leases. Since the lease is added to the array in the order of application, it is possible to avoid the expiration of a single application in a random manner.

  • Line 39: The current time passed in generates randomness for the seed, avoiding the pseudo-randomness of Java. stay Why are all random numbers in Java pseudorandom? " Detailed analysis.
  • Lines 41 to 43: Randomly swap the following elements to the current position (i).

• Line 50: Call the # internalCancel() method, the lease that has expired offline, in "Eureka Source Parsing - Application Case Registration Discovery (IV) Self-Protection Mechanism", "3.2 Offline Application Example Information" Detailed analysis.

666. eggs

(iv) An article that was originally thought to be relatively easy consumed more time than imagined, perhaps four hours. The main card is compensation time, which is not understood at present. If you know a fat friend, please let me know.

Fat friend, how nice is it to share my public number (taro source code) with your fat friend?