Micrometer for Prometheus monitoring supports multiple endpoint URL s

Keywords: Java Spring github

1, background

When using Prometheus as a monitoring system (java), the general practice is that the system exposes the endpoint URL to prometheus, such as / metrics, and then Prometheus pulls the index data from the url.
The main uses are spring-boot-starter-actuator, micrometer-registry-prometheus.
But the problem is that the default exposed endpoint is / prometheus, the full path is / actuator/prometheus, and there is only one URL, so what if there is such a scenario?

  • prometheus limits the number of metrics in a single URL to no more than 1W
  • For business metrics monitoring, each business side wants to use different URL exposure metrics (it should not want to put all business metrics in one URL)

So you need to add multiple URL s. What should you do?

2, practice

Looking at the micrometer source (mainly Prometheus Metrics Export AutoConfiguration, etc.), you can see the initialization process for the default endpoint prometheus.
Haven't found that other open API s can easily add and expose multiple endpoint URL s, so you can copy his process and write a set of configurations by yourself. After practice, the configurations written by yourself will not be too many.
For example, if I want to expose an apple endpoint, the URL of / actuator/apple, the code is as follows:

The first is to define endpoints

@WebEndpoint(id = "apple")
public class AppleScrapeEndPoint {

    private final CollectorRegistry collectorRegistry;

    public AppleScrapeEndPoint(CollectorRegistry collectorRegistry) {
        this.collectorRegistry = collectorRegistry;
    }

    @ReadOperation(produces = TextFormat.CONTENT_TYPE_004)
    public String scrape() {
        try {
            Writer writer = new StringWriter();
            TextFormat.write004(writer, this.collectorRegistry.metricFamilySamples());
            return writer.toString();
        } catch (IOException ex) {
            // This actually never happens since StringWriter::write() doesn't throw any
            // IOException
            throw new RuntimeException("Writing metrics failed", ex);
        }
    }
}

Then we define the AppleProperties ConfigAdapter

public class ApplePropertiesConfigAdapter extends PropertiesConfigAdapter<PrometheusProperties>
        implements PrometheusConfig {

    ApplePropertiesConfigAdapter(PrometheusProperties properties) {
        super(properties);
    }

    @Override
    public String get(String key) {
        return null;
    }

    @Override
    public boolean descriptions() {
        return get(PrometheusProperties::isDescriptions, PrometheusConfig.super::descriptions);
    }

    @Override
    public Duration step() {
        return get(PrometheusProperties::getStep, PrometheusConfig.super::step);
    }

}

Finally, configuration initialization

@Configuration
@AutoConfigureAfter(value = {PrometheusMetricsExportAutoConfiguration.class})
@ConditionalOnClass(value = {PrometheusMeterRegistry.class})
@ConditionalOnProperty(prefix = "management.metrics.export.apple", name = "enabled", havingValue = "true",
        matchIfMissing = true)
public class ApplePrometheusAutoConfiguration {

    @Bean(name = "applePrometheusProperties")
    @ConfigurationProperties(prefix = "management.metrics.export.apple")
    public PrometheusProperties applePrometheusProperties() {
        return new PrometheusProperties();
    }

    @Bean(name = "applePrometheusConfig")
    public PrometheusConfig applePrometheusConfig() {
        return new ApplePropertiesConfigAdapter(applePrometheusProperties());
    }

    @Bean(name = "appleMeterRegistry")
    public PrometheusMeterRegistry appleMeterRegistry(Clock clock) {
        return new PrometheusMeterRegistry(applePrometheusConfig(), appleCollectorRegistry(), clock);
    }

    @Bean(name = "appleCollectorRegistry")
    public CollectorRegistry appleCollectorRegistry() {
        System.out.println("=======appleCollectorRegistry");
        return new CollectorRegistry(true);
    }

    @Configuration
    @ConditionalOnEnabledEndpoint(endpoint = AppleScrapeEndPoint.class)
    public static class TicketScrapeEndpointConfiguration {

        @Resource
        private CollectorRegistry appleCollectorRegistry;

        @Bean(name = "appleEndpoint")
        @ConditionalOnMissingBean
        public AppleScrapeEndPoint appleEndpoint() {
            return new AppleScrapeEndPoint(appleCollectorRegistry);
        }

    }

}

Then configure the newly added endpoint in the configuration file

management:
  endpoint:
    prometheus:
      # Close the default prometheus endpoint and create your own
      enabled: false
    health:
      show-details: always
  endpoints:
    web:
      exposure:
        include: ['health', 'apple']
        

This is done, and you can see your newly added URL in the / actuator URL.
If you want to add more than one copy, just change the name according to the above copy code, and don't forget to add it in the configuration file include.

This basically solves the problem, but looks uncomfortable. I have multiple URL s and need COPY copies of this code, and it's almost the same, so we can consider active configuration.
Reduce duplicate code creation, as follows:

For example, I would like to add another endpoint of a, namely / actuator/a.

The first is to define endpoints:

@Component
@DatagridEndpoint
@WebEndpoint(id = "a")
public class AEndpoint {

    private CollectorRegistry collectorRegistry;
    public AEndpoint(){
    }

    @ReadOperation(produces = TextFormat.CONTENT_TYPE_004)
    public String scrape() {
        try {
            Writer writer = new StringWriter();
            TextFormat.write004(writer, this.collectorRegistry.metricFamilySamples());
            return writer.toString();
        } catch (IOException ex) {
            // This actually never happens since StringWriter::write() doesn't throw any
            // IOException
            throw new RuntimeException("Writing metrics failed", ex);
        }
    }
}

Then there is the configuration process.

@Slf4j
@Component
@AutoConfigureAfter(value = {PrometheusMetricsExportAutoConfiguration.class})
@ConditionalOnClass(value = {PrometheusMeterRegistry.class})
public class MetricsExportAutoConfiguration implements BeanDefinitionRegistryPostProcessor,
        ApplicationContextAware {

    private ApplicationContext applicationContext;

    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }

    public class AutoPropertiesConfigAdapter extends PropertiesConfigAdapter<PrometheusProperties>
            implements io.micrometer.prometheus.PrometheusConfig {

        AutoPropertiesConfigAdapter(PrometheusProperties properties) {
            super(properties);
        }

        @Override
        public String get(String key) {
            return null;
        }

        @Override
        public boolean descriptions() {
            return get(PrometheusProperties::isDescriptions, io.micrometer.prometheus.PrometheusConfig.super::descriptions);
        }

        @Override
        public Duration step() {
            return get(PrometheusProperties::getStep, PrometheusConfig.super::step);
        }

    }

    @Override
    public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry beanDefinitionRegistry) throws BeansException {
    }

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory factory) throws BeansException {

        Map<String, Object> beansMap = applicationContext.getBeansWithAnnotation(DatagridEndpoint.class);
        if (CollectionUtils.isEmpty(beansMap)) {
            return;
        }

        Clock clock = applicationContext.getBean(Clock.class);
        Preconditions.checkNotNull(clock);

        for (Map.Entry<String, Object> entry : beansMap.entrySet()) {
            Object bean = entry.getValue();
            WebEndpoint webEndpoint = bean.getClass().getAnnotation(WebEndpoint.class);
            if (null == webEndpoint) {
                continue;
            }
            String endPointName = webEndpoint.id();
            if (Strings.isNullOrEmpty(endPointName)) {
                continue;
            }
            // prometheus properties bean
            BeanDefinitionBuilder prometheusPropertiesBeanDefinitionBuilder = BeanDefinitionBuilder
                    .genericBeanDefinition(PrometheusProperties.class);
            BeanDefinition prometheusPropertiesBeanDefinition = prometheusPropertiesBeanDefinitionBuilder.getRawBeanDefinition();
            ((DefaultListableBeanFactory) factory).registerBeanDefinition(endPointName + "PrometheusProperties", prometheusPropertiesBeanDefinition);

            PrometheusProperties prometheusProperties = applicationContext.getBean(endPointName + "PrometheusProperties", PrometheusProperties.class);

            // prometheus config bean
            BeanDefinitionBuilder prometheusConfigBeanDefinitionBuilder = BeanDefinitionBuilder
                    .genericBeanDefinition(AutoPropertiesConfigAdapter.class, () -> new AutoPropertiesConfigAdapter(prometheusProperties));
            BeanDefinition prometheusConfigBeanDefinition = prometheusConfigBeanDefinitionBuilder.getRawBeanDefinition();
            ((DefaultListableBeanFactory) factory).registerBeanDefinition(endPointName + "PrometheusConfig", prometheusConfigBeanDefinition);

            // collector registry bean
            BeanDefinitionBuilder collectorRegistryBeanDefinitionBuilder = BeanDefinitionBuilder
                    .genericBeanDefinition(CollectorRegistry.class, () -> new CollectorRegistry(true));
            BeanDefinition collectorRegistryBeanDefinition = collectorRegistryBeanDefinitionBuilder.getRawBeanDefinition();
            ((DefaultListableBeanFactory) factory).registerBeanDefinition(endPointName + "CollectorRegistry", collectorRegistryBeanDefinition);

            PrometheusConfig prometheusConfig = applicationContext.getBean(endPointName + "PrometheusConfig", AutoPropertiesConfigAdapter.class);
            CollectorRegistry collectorRegistry = applicationContext.getBean(endPointName + "CollectorRegistry", CollectorRegistry.class);

            // prometheus meter registry bean
            BeanDefinitionBuilder meterRegistryBeanDefinitionBuilder = BeanDefinitionBuilder
                    .genericBeanDefinition(PrometheusMeterRegistry.class, () -> new PrometheusMeterRegistry(prometheusConfig, collectorRegistry, clock));
            BeanDefinition meterRegistryBeanDefinition = meterRegistryBeanDefinitionBuilder.getRawBeanDefinition();
            ((DefaultListableBeanFactory) factory).registerBeanDefinition(endPointName + "MeterRegistry", meterRegistryBeanDefinition);

            Reflect.on(bean).set("collectorRegistry", collectorRegistry);

        }
    }
}

Finally, add exposed endpoints to include in the configuration file

management:
  endpoint:
    prometheus:
      # Close the default prometheus endpoint and create your own
      enabled: false
    health:
      show-details: always
  endpoints:
    web:
      exposure:
        include: ['health', 'apple', 'a']
        

ok, next time you want to add extra, you just need to create the same class as endpoint a, change the id value, and then expose it in the include in the configuration file.
The Metrics Export AutoConfiguration class automatically creates other configurations without duplicating code.

Oh, yeah, the annotation about Datagrid Endpoint is just a simple annotation, as follows

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface DatagridEndpoint {
}

That's it.
Of course, the above is just scrawl code, you can see their own changes, more suitable for their own project!

See: https://github.com/kute/prome...

Questions and timely contact

Posted by Maeltar on Fri, 11 Oct 2019 02:25:08 -0700